Abstract
Introduction Rapid antigen detection tests (RDT) are suitable for large-scale testing for SARS-CoV-2 among the population and recent studies have shown that self-testing with RDT in the general population is feasible and yields acceptable sensitivities with high specificity. We aimed to determine the accuracy of two different RDT’s, with two different sample collection methods for one of the RDT’s among healthcare workers (HCW). Secondary objectives were to determine the accuracy of RDT using a viral load cut-off as proxy of infectiousness and to identify predictors for a false negative RDT.
Methods Centers that participated were secondary care hospitals, academic teaching hospitals, and long-term care facilities. All HCW that met inclusion criteria were asked to perform a RDT self-test next to a regular SARS-CoV-2 nucleic acid amplification test (NAAT). Three study groups were created. Study group 1; Veritor(tm) System, Becton Dickinson, Franklin Lakes, USA (BD-RDT) with combined oropharyngeal - mid-turbinate nasal sampling, group 2; BD-RDT with mid-turbinate nasal sampling only and group 3; SD Biosensor SARS-CoV-2 Rapid Antigen Test, Roche, Basel, Switzerland (Roche-RDT) with combined oropharyngeal - mid-turbinate nasal sampling. RDT accuracy was calculated using NAAT as reference standard. For samples processed in the cobas® 6800/8800 platform (Roche Diagnostics, Basel, Switzerland), established cycle threshold values (Ct-values) could be converted into viral loads. A viral load cut-off of ≥5.2 log10 SARS-CoV-2 E gene copies/ml was used as proxy of infectiousness. Logistic regression analysis was performed to identify predictors for a false negative RDT.
Results In total, 7,196 HCW were included. Calculated sensitivities were 61.5% (95%CI 56.6%-66.3%), 50.3% (95%CI 42.8%-57.7%) and 74.2% (95%CI 66.4%-80.9%) for study groups 1, 2 and 3, respectively. After application of a viral load cut-off as a proxy for infectiousness for samples processed in the cobas® 6800/8800 platform sensitivities increased to 82.2% (95%CI 76.6-86.9%), 61.9% (95%CI 48.8%-73.9%) and 90.2% (95%CI 76.9%-97.3%) for group 1, group 2 and group 3, respectively. Multivariable regression analysis showed that use of Roche-RDT (p <0.01), combined oropharyngeal - mid-turbinate nasal sampling (p <0.05) and the presence of COVID-19 like symptoms at the time of testing (p <0.01) significantly reduced the likeliness of a false-negative RDT result.
Conclusion SARS-CoV-2 RDT has proven able to identify infectious individuals, especially when upper respiratory specimen is collected through combined oropharyngeal - mid-turbinate sampling. Reliability of self-testing with RDT among HCW seems to depend on the type of RDT, the sampling method and the presence of COVID-19 like symptoms at the time of testing.
Introduction
Nucleic acid amplification testing (NAAT) is considered to be the reference test method for detection of SARS-CoV-2. Although less sensitive than NAAT, rapid antigen detection tests (RDT) using lateral flow assay technology are suitable for large-scale testing for SARS-CoV-2 of individuals among the general population presenting with COVID-19 like symptoms (1-5). Recent studies have shown that self-testing with RDT is feasible and yields acceptable sensitivities with high specificity (6-8). Self-testing has the potential for frequent large-scale use at a relatively low cost.
We aimed to determine the accuracy of two different RDT’s, with two different sample collection methods for one of the RDT’s. Secondary objectives were to determine the accuracy of RDT using a viral load cut-off as proxy of infectiousness and to identify predictors for a false negative RDT. Tertiary objectives were to assess the practical applicability of RDT among HCW when conducting the test themselves and the competence of HCW to interpret the RDT result themselves.
Methods
Study design and participants
We conducted a multicenter prospective cohort study among HCW with COVID-19 like symptoms who were analyzed for SARS-CoV-2 infection at their health care center according to local test policy. Centers that participated were secondary care hospitals, academic teaching hospitals, and long-term care facilities. The study was conducted between October 31st, 2020, and February 2nd, 2021. During this period circulation of Alpha, Delta or Omicron variants of SARS-CoV-2 was not yet observed in the Netherlands, according to national surveillance data (9).
For this study three study groups were defined. At the start of the study, all centers who were participating at that time were assigned to study group 1. Subsequent to inclusion of the required number of participants in study group 1 in accordance with the power calculation, the participating centres were switched to either study group 2 or 3. Centres were free to choose in which consecutive study group (2 or 3) they would participate. In study group 1, the BD-RDT (BD Veritor(tm) System for Rapid Detection of SARS-CoV-2, Becton Dickinson company, Franklin Lakes, USA) was used with combined oropharyngeal - mid-turbinate sampling. In study group 2, the BD-RDT was combined with mid-turbinate nasal sampling only. In study group 3, the Roche-RDT (SD Biosensor SARS-CoV-2 Rapid Antigen Test, Roche Diagnostics, Basel, Switzerland) was used with combined oropharyngeal - mid-turbinate nasal sampling.
All employees of the participating healthcare centers, both medically trained and non-medically trained, were designated as HCW. Following local test policy, all HCW with COVID-19 like symptoms made an appointment at their healthcare center of employment for a SARS-CoV-2 test based on NAAT. Every HCW aged 18 years or older and able to understand the written instructions in Dutch, was asked to participate and provide verbal informed consent.
Sampling and self-test procedure
A sample for SARS-CoV-2 NAAT was obtained by a trained professional using the sampling method that was routinely used at the participating healthcare center. For the NAAT, nasopharyngeal or mid-turbinate sampling combined either with or without oropharyngeal sampling was assessed. Study subjects were handed a package containing the RDT device, swab stick, test medium cup, small clear plastic zip lock bag, instruction forms and question and result form. The zip lock bag was added to store the test cassette during waiting time to minimize risk of virus transmission while still allowing visual assessment of the test. Study subjects were referred to a dedicated room where they performed the self-test following the instructions. The self-test area was equipped with a mirror, table, plasticized instructions, holder for the test medium cup, chair, (chemical) waste bin, disinfection wipes, hand alcohol and sanitizer. The participants performed the tests alone, without external aid. For the instruction, question and result forms, see Supplementary Methods 1 and 2.
After conducting the self-test, a timer was to be set to 15 minutes and the short questionnaire was to be filled in. After 15 minutes the study subject was to review the test result, fill out the result on the form and take a picture of the RDT in the plastic bag, the personal barcode and the self-assessed result. The picture was to be uploaded to an anonymized and secured online cloud (Box Inc., Redwood City, USA) by scanning a QR code. The filled-out questionnaire and result form were to be deposited at the return point. A negative RDT result had no consequence on quarantine measures. In case of a positive RDT result, the same local measures applied as with a positive NAAT result.
Nucleic Acid Amplification Tests platforms and viral load cut-off
The method of collection, transport and storage conditions of the samples were in accordance with the local protocols used in the participating centers and could vary between participating centers. Participating centers made use of their routine diagnostic laboratory protocols. Although NAAT was used as reference standard in all laboratories, the specific techniques, platforms and assays were different (Table 1). Only for real-time quantitative reverse transcription PCR (qRT-PCR) results of samples processed in the cobas® 6800/8800 platform (Roche Diagnostics, Basel, Switzerland), established cycle threshold values (Ct-values) could be converted into viral loads (Supplementary Methods 3). A viral load cut-off of ≥5.2 log10 E gene copies/ml was used as proxy of infectiousness (10).
Sample size
We assumed the diagnostic accuracy of RDT to be lower than when performed by professionals, and based the sample size calculation on an expected sensitivity of 80%, with a margin of error of 7%, type I error of 5% and power of 90% (6). Hence, the minimum number of participants with a positive NAAT result was 125 per study group. The NAAT result positivity percentage was monitored over time and recruitment was adjusted if needed.
Data collection and statistical analysis
Pseudonymized subject data and results of both the RDT and NAAT were collected in Castor EDC (Castor, Amsterdam, the Netherlands). Subject data consisted of sex, age, group of profession (medically trained or non-medically trained personnel), present symptoms and their answers to questions about the practicality of the test. If the RDT result and NAAT result were incongruent, the uploaded photo was assessed by a member of the research team, to acknowledge the test interpretation made by the study subject. The calculations and analyses were based on RDT results reported by the HCW.
Calculations and statistical analysis were performed using SPSS version 24 (IBM Corp. Armonk NY, USA). The overall sensitivity and specificity of RDT in the 3 study groups were calculated. When the reference NAAT was a qRT-PCR performed on the cobas® 6800/8800 platform, sensitivity and specificity of the RDT could be adjusted for infectiousness. Univariable and multivariable logistic regression analyses were performed to examine whether following variables were independently associated with a false negative result in RDT as compared to NAAT: study group (categorical variable), center category (categorical variable), sex (categorical variable), age (continuous variable), group of profession (categorical variable) and current COVID-19 related symptoms (categorical variable). Variables were included in the multivariable analysis when p <0,2 in the univariable analysis. Invalid RDT results were excluded when determining sensitivity and specificity of self-testing and interpreted as not false negative when determining the variables associated with a false negative RDT result.
Ethics
The planning, conduction, documentation and reporting of the study were in line with the Declaration of Helsinki, as revised in 2013. The study protocol was reviewed by the Dutch ‘Medical research Ethics Committees United’ (MEC-U) and was judged to be beyond the scope of the Dutch medical scientific research act (WMO) (MEC-U subject: W20.250). A waiver of written informed consent was granted as handling of documents obtained from possibly infectious participants was considered a potential safety hazard.
Results
Study design and baseline characteristics
During the study period, 7,495 HCW were considered eligible for participation of whom 7,196 HCW were included (Fig. 1). Participants were excluded if there was a missing RDT result (3%), NAAT result (<1%), study ID (<1%) or date of testing (<1%). In study group 1, BD-RDT with combined oropharyngeal - mid-turbinate nasal sampling, 3,255 HCW were included. In study group 2, BD-RDT with mid-turbinate nasal sampling only, 1,729 HCW were included. In study group 3, Roche-RDT with combined oropharyngeal - mid-turbinate nasal sampling, 2,212 HCW were included. Baseline characteristics of the three study groups were comparable between all groups and are presented in Table 2.
*BD-RDT with combined oropharyngeal - mid-turbinate nasal sampling. ** BD-RDT with mid-turbinate nasal sampling only.
***Roche-RDT with combined oropharyngeal - mid-turbinate nasal sampling. Abbrevations: RDT, rapid antigen detection test; NAAT, nucleic acid amplification test.
An overview of the participating centers, corresponding number of inclusions in the various study groups are presented in Table 3. All but two centers started in study group 1 and then decided to proceed to study group 2 or 3 or to stop participation. One center started in study group 2, one center in study group 3 and one center decided to stop further participation after the required numbers in study group 1 had been reached.
RDT sensitivity and specificity
In study group 1, 411 (12.6%) participants had a NAAT confirmed SARS-CoV-2 infection and 253 (7.8%) participants had a positive result in the RDT. RDT sensitivity and specificity were 61.5% (95%CI 56.6% to 66.3%) and 99.9% (95%CI 99.6% to 100%), respectively (Table 4). In study group 2, 191 (11.0%) participants had a NAAT confirmed SARS-CoV-2 infection and 96 (5.6%) participants had a positive result in the RDT resulting in a sensitivity and specificity of 50.3% (95%CI 42.8% to 57.7%) and 99.7% (95%CI 99.3% to 99.9%), respectively. In study group 3, 152 (6.9%) participants had a NAAT confirmed SARS-CoV-2 infection and 119 (5.4%) participants had a positive result in the RDT resulting in a sensitivity of 74.2% (95%CI 66.4% to 80.9%) and specificity of 99.7% (95%CI 99.3% to 99.9%).
For qRT-PCR results of samples processed in the cobas® 6800/8800 platform, a viral load cut-off was used as a proxy for infectiousness. In study group 1, 304 (13.8%) of all 2,210 samples processed on the cobas® 6800/8800 platform had a positive qRT-PCR result and 233 (76.6%) of these, had a viral load above the cut-off. This resulted, after application of a viral load cut-off as a proxy for infectiousness, in a sensitivity and specificity of 82.2% (95%CI 76.6 to 86.9%) and 99.3% (95%CI 98.8% to 99.6%) for the BD-RDT with combined oropharyngeal - mid-turbinate nasal sampling, respectively (Table 5). The same procedure was applied for study group 2 and 3. In study group 2, 86 (12.6%) of all 1,992 samples run on the cobas® 6800/8800 platform had a positive qRT-PCR result and of those, 65 (75.6%) had a viral load above the cut-off. After application of a viral load cut-off as a proxy for infectiousness, this resulted in a sensitivity and specificity of 61.9% (95%CI 48.8% to 73.9%) and 99.8% (95%CI 99.1% to 100%) for the BD- RDT with mid-turbinate nasal sampling only, respectively. In study group 3, 56 (5.6%) of all 997 samples run on the cobas® 6800/8800 platform had a positive qRT-PCR result and of those, 41 (73.2%) had a viral load above the cut-off. This resulted, after application of a viral load cut-off as a proxy for infectiousness, in a sensitivity and specificity of 90.2% (95%CI 76.9% to 97.3%) and 99,4% (95%CI 99.1% to 99.6%) for the Roche-RDT with combined oropharyngeal - mid-turbinate nasal sampling, respectively.
Multivariable analysis
The relation between a false-negative RDT (negative RDT result in case of a positive NAAT, irrespective of viral load) was assessed for seven variables by conducting a univariable logistic regression analysis (Table 6). Out of these seven variables, three were associated with the occurrence of a false-negative RDT result with a p-value <0.2 and were included in the multivariable analysis (Table 6). The type of RDT was independently associated with the occurrence of a false-negative RDT result (p <0,01). Participants that assessed a Roche-RDT were significantly less likely to have a false-negative RDT result compared to participants that assessed a BD-RDT. Also, the method of sampling was independently associated with the occurrence of a false-negative RDT result (p <0,05). Participants that performed combined oropharyngeal – mid-turbinate nasal sampling were significantly less likely to have a false-negative RDT result compared to participants that performed mid-turbinate nasal sampling only. As the third variable, the presence of COVID-19 like symptoms at the time of testing was independently associated with the non-occurrence of a false-negative RDT result (p <0.01).
RDT assessment and congruency with NAAT
Of all included participants, 400 (5.6%) subjects had an incongruent NAAT and RDT result, i.e. a positive NAAT result with a negative or invalid RDT result, or a negative NAAT result with a positive or invalid RDT result. Of those, 266 (66.5%) participants uploaded a photo of their RDT to the cloud. In 25 (9.4%) out of a total of 266 uploaded photos of incongruent results, the RDT was assessed differently by the reviewer than the HCW. Increasing HCW age was associated with reading errors among all incongruent test results (p 0,03).
Questionnaire data
Additional data concerning the questionnaire of the included participants are shown in Table 7. Medians in all three study groups are similar for the perceived difficulty of the RDT. The least doubt about the RDT is observed in study group 3. Participants were most likely to use the test again when they had COVID-19 like symptoms or recommend the test to a colleague in study group 2 and study group 3.
Discussion
SARS-CoV-2 rapid antigen tests performed by HCW as self-tests were able to detect most infectious COVID-19 cases. The results were better when combined oropharyngeal - mid-turbinate samples were used compared to mid-turbinate samples only. Also, the sensitivity varied between different brands of RDT. This study shows that RDT’s can be used as self-tests but that there are certain limitations considering the sensitivity.
In general, RDT’s perform well in individuals with a high SARS-CoV-2 viral load which is usually present in the pre-symptomatic and early symptomatic phase – the first 5-7 days – of COVID-19 (11,12). Multiple validation studies for RDT have been published in which a higher sensitivity is observed at low Ct-values (high viral loads) for different RDT brands and different qRT-PCR platforms (1-3, 13-17). It is a point of discussion above which viral load and therefore below which Ct-value people are likely to be infectious. After infection with SARS-CoV-2, viral RNA can be detected by qRT-PCR during a long time, even if the infected person is no longer infectious (18, 19). Taking the viral load of SARS-CoV-2 as a proxy for infectiousness – by determining the viral load cut-off above which 95% of qRT-PCR positive samples had a positive virus culture – and extrapolating it to a cut-off Ct-value for comparable PCR platforms is the most feasible and closest to the true rate of infectiousness (6, 10, 20).
When adjusted for infectiousness for all qRT-PCR results of samples processed in the cobas® 6800/8800 the sensitivity clearly increased for each group. The sensitivities were 82,2%, 61,9% and 90,2% for study group 1, study group 2 and study group 3, respectively. The increased sensitivity after adjustment for infectiousness is in line with previous studies (6, 10). Specificity decreased slightly in all study groups after adjustment for infectiousness. This is a consequence of a number of previously true-positive RDT results compared with qRT-PCR results with viral loads below the cut-off that now pertained to the false-positive group. As WHO stated, RDTs would need a sensitivity ≥80% and a specificity ≥97% (21). Both BD-RDT and Roche-RDT with combined oropharyngeal - mid-turbinate sampling meet this requirement and BD-RDT in the case of mid-turbinate sampling only does not meet this WHO recommendation on sensitivity. Addition of an oropharyngeal swab to a mid-turbinate nasal sample clearly improves the yield.
We found that reliability of self-testing with RDT among HCW seems to depend on the type of RDT, sampling method of the mucosal specimen and the presence of COVID-19 like symptoms at the time of testing. Participants that assessed a Roche-RDT were significantly less likely to have a false-negative RDT result compared to participants that assessed a BD-RDT (p <0.01). Subjects that performed combined oropharyngeal – mid-turbinate nasal sampling were significantly less likely to have a false-negative RDT result compared to participants that performed mid-turbinate nasal sampling only (p <0,05). By contrast, it appears from the literature that nasal mid-turbinate self-sampling versus nasopharyngeal sampling performed by a trained subject would not have an effect on sensitivity of an RDT (7, 22). To our knowledge, this is the first clinical study that has shown a significant difference in the reliability of the RDT type and sampling method when applied as self-test. Also, the presence of COVID-19 like symptoms at the time of testing appeared to be a factor in the multivariable regression analysis that reduces the probability of a false-negative RDT (p <0,01). This is consistent with the finding of Thirion-Romero et al. that the presence of symptoms was a predictor of positivity for the Panbio RDT (23).
Our study was conducted in the Netherlands in a time frame just before a transition from the original Wuhan strain to the Alpha variant (9). Recent data of the FDA suggested that RDT do detect the omicron variant but may have reduced sensitivity (24). In particular, the media raised the question whether this alleged reduced sensitivity of the RDT in the SARS-CoV-2 Omicron variant might be due to the sampling method. Omicron is said to be more detectable by an RDT with a combined oropharyngeal - nasal sampling than a nasal sampling only. Our study shows that a significant difference in RDT sensitivity is demonstrable with a difference in the sampling method. Whether the size of this difference is applicable with the emergence of the omicron variant is still to be determined.
Self-test RDT accuracy among HCW is not affected by reading errors, nor does it seem to matter for the performance of the test whether someone is medically trained or not. In our study hardly any self-reading errors were made. The total percentage of reading errors of all inclusions is estimated to be 0.5%, assuming that all congruent results have been read correctly and that the percentage of reading error in the group of incongruent results with no photo uploaded is also 9.5%. Little has been described in literature about self-reading of antigen tests in general. It can be assumed that HCW may have more experience with the interpretation of these types of tests than e.g. laypersons. Lindner et al. reported that the inter-rater reliability in a double reading of an RDT between laypersons and professionals was very high (kappa 0,98) (8) and Cassuto et al stated that interpretation errors in RDT also occur very rarely in laymen (25).
Our study had a number of limitations. First was the number of different NAAT platforms used. Except for the cobas® 6800/8800 platform the correlation between Ct-value produced by qRT-PCR platforms and the SARS-CoV-2 viral load is not known. Further, not every NAAT platform renders a Ct-value as a result and for that we were unable to extrapolate those NAAT results to viral loads. As a result, we had to omit a large amount of subjects from our logistic regression analysis. We could only take the data that was run on the cobas® 6800/8800 platform, because for this platform we previously determined the correlation between viral load and Ct-value (10). In future research, it would be advisable - in the case of a multicenter study - to have a known and described cut-off point for each platform used or to run all samples primarily on a single platform to facilitate this comparison process. The most accurate would be to perform virus cultures, although this will not always be feasible. Secondly, different methods of upper respiratory specimen collection, as nasopharyngeal or mid-turbinate either with or without oropharyngeal sampling for NAAT were included when performed by trained personnel, according to the local protocol in de participating center which makes the comparison of results of the reference test between the centers not completely reliable. Thirdly, the experience with the test procedure may grow over time leading to a better performance for tests performed at the end of the study. As the procedures were not performed in the same period this may play a role. However, this may be of limited importance as the results of the BD-RDT in period two were worse than in period one.
In conclusion, SARS-CoV-2 rapid antigen tests have proven to be able to detect infectious individuals when upper respiratory specimen is collected with a combined oropharyngeal - mid- turbinate sampling method. Reliability of self-testing with RDT among HCW seems to depend on the type of RDT, sampling method and the presence of COVID-19 like symptoms at the time of testing. Self-test RDT accuracy among HCW is not affected substantially by reading errors, nor whether or not a HCW is medically trained. This makes self-testing using RDT a useful tool to detect infectious individuals.
Data Availability
All data produced in the present study are available upon reasonable request to the authors.
Acknowledgement
We would like to thank Ons Tweede Thuis for participating in the study. I would also like to thank Berit Hol for the effort put into the instruction forms.
References
