Accuracy of COVID-19 self-tests with unsupervised nasal or nasal plus oropharyngeal self-sampling in symptomatic individuals in the Omicron period

Introduction

Rapid antigen diagnostic tests (Ag-RDTs) show promising performance for SARS-CoV-2 infection detection.^1-5 Ag-RDTs require minimal equipment, provide a result within 15-30 minutes, and can be performed in a range of settings without laboratory facilities. While Ag-RDTs were initially introduced for use by trained professionals, they are currently widely available over the counter. Self-testing, without supervision of a trained professional, lowers the threshold for testing and allows individuals to obtain a test result quickly, at their own convenience. This in turn could support early detection and self-isolation of infectious cases and reduce community transmission.⁶

We previously showed that the SD Biosensor by Roche Diagnostics Ag-RDT with unsupervised nasal self-sampling had a 78.5% sensitivity in symptomatic individuals.⁷ However, since end November 2021, the Omicron variant rapidly replaced the Delta variant. Performance of Ag-RDTs for Omicron might be different due to alterations in viral proteins and infection dynamics. Initial studies comparing Omicron and Delta variants found similar sensitivities for molecular tests⁸, mixed analytical performance of lateral flow devices^9,10, and similar real-world sensitivities for Ag-RDTs with sampling and testing by trained professionals.^11,12 Additionally, anecdotal concerns were raised about the performance of Ag-RDTs when applying nasal self-sampling only since Omicron variant’s viral particles seem more prevalent in the throat than nose. One study indeed showed improved Ag-RDT sensitivity with combined throat and nasal sampling by trained professionals.¹² Currently, real-world data on comparative accuracy of Ag-RDTs with unsupervised nasal or combined oropharyngeal plus nasal (OP-N) self-sampling are lacking.

Therefore, we studied the accuracy of three widely commercially available Ag-RDTs with unsupervised self-sampling during and after emergence of Omicron using RT-PCR as reference standard; to evaluate whether accuracies of Ag-RDTs with nasal self-sampling changed over time; and to quantify whether addition of oropharyngeal to nasal self-sampling yields higher diagnostic accuracies.

Methods

The study is reported according to the STARD 2015 guidelines.¹³

Outcomes and statistical analyses

Primary outcomes were diagnostic accuracy (sensitivity, specificity, positive and negative predictive values with corresponding 95% confidence intervals [95% CI]) of each Ag-RDT either with nasal or with OP-N self-sampling, and RT-PCR testing as reference. Secondary outcomes were diagnostic accuracies stratified by reason for testing (confirmatory testing after a positive self-test at one’s own initiative, type of symptoms, close contact, or other reason), COVID-19 vaccination status (no vaccination or vaccinated once, twice, or thrice), having had prior SARS-CoV-2 infection, gender, and age (16-40, >40 years).

Also, we assessed whether performance of the three Ag-RDTs with nasal self-sampling changed over time during emergence of Omicron, by analysing the Ag-RDTs’ sensitivities and specificities per week. Weekly intervals were chosen because the share of the Omicron variant in The Netherlands was assessed weekly in the national pathogen surveillance.¹⁶ Sensitivities in the first and last week were compared by Chi-square tests.

All primary and secondary diagnostic accuracies were also determined after applying a viral load cut-off (≥5.2 log10 SARS-CoV-2 E-gene copies/mL). This was the viral load cut-off above which 95% of people with a positive RT-PCR test had a positive virus culture based on previous work.² Furthermore, considering the large influence of confirmatory testers in our study populations, all analyses were repeated stratified by confirmatory testing (yes/no).

Finally, self-reported user experiences with each Ag-RDT and self-reported numbers of infections that may have been missed by baseline RT-PCR testing were assessed.

We performed complete cases analysis because the number of individuals without RT-PCR test or Ag-RDT results was very low (Figure 1).

• Download figure
• Open in new tab

Figure 1.

Flow of study participants in the omicron period, presented separately for nasal (top) and oropharyngeal plus nasal sampling (bottom). Flowflex = the Acon Labs Flowflex COVID-19 Antigen Home Test; MPBio = the MP Biomedicals Rapid SARS-CoV-2 Antigen Test Card; Clinitest = the Siemens-Healthineers CLINITEST Rapid COVID-19 Antigen Test. “Ag-RDT inconclusive” is a combination of Ag-RDTs that showed no control line, test tubes that were dropped, an Ag-RDTs that provided a result participants had difficulties to interpret (e.g., very light line at the “T”).

Results

A total of 3,076 individuals participated in the Delta-Omicron transition phase prior to phase-1 (Figure S1) and a further 2,199 in phase-1 and 1,222 individuals in phase-2 (Figure 1). Participant characteristics for the Delta-Omicron transition phase are presented in Table S1 and for phases 1 and 2 in Table 1.

Overall Ag-RDT accuracies with nasal self-sampling in the Omicron period

Overall sensitivities were 79.0% (74.7-82.8%) for Flowflex, 69.9% (65.1-74.4%) for MPBio, and 70.2% (65.6-74.5%) for Clinitest (Table 2, Figure 2). After applying the viral load cut-off, sensitivities increased to 85.6% (81.5-89.1%), 78.5% (73.8-82.8%), and 77.0% (72.4-81.2%), respectively (Figure S2). Specificities were >92%, positive predictive values >94%, and negative predictive values >59% for all three Ag-RDTs in all analyses (Table 2), with slightly higher specificities and PPVs for MPBio and Clinitest, and higher NPVs for Flowflex.

View this table:

• View inline
• View popup

Table 2. Diagnostic accuracy parameters for the three Ag-RDTs in symptomatic individuals in the Omicron period. Values are percentages (95% confidence interval) unless stated otherwise.

• Download figure
• Open in new tab

Figure 2.

Sensitivities with 95% confidence intervals of the antigen rapid test-RT-PCR reference standard test comparisons stratified according to COVID-19 vaccination status, previous infection status, sex, and age, with nasal sampling for all three tests (top) and oropharyngeal plus nasal sampling for MPBio and Clinitest (bottom). The vertical line indicates the sensitivity of the Ag-RDT in the respective overall study population, and the number of positive RT-PCR tests out of the total or subgroup between parentheses.

Supplementary Tables S2-S4 show all 2×2 tables.

Ag-RDT accuracy changes over time during emergence of Omicron with nasal self-sampling only

Sensitivities of all three Ag-RDTs were highest during the first week (Figure 3) when the Omicron share was 28.6%: 87.0% (79.7-92.4%) for Flowflex, 80.0% (51.9-95.7%) for MPBio, and 83.1% (72.9-90.7%) for Clinitest. With emergence of Omicron, sensitivities decreased by 6.1 percentage points for Flowflex (Chi-square test statistic 2.0; p-value=0.16), 7.0 for MPBio (Chi-square test statistic 0.28; p-value=0.60), and 12.8 for Clinitest (Chi-square test statistic 5.0; p-value=0.025). Specificities varied between 93.2% and 99.6% over time. Application of the viral load cut-off resulted in higher sensitivities but all trends over time remained similar (Figure 3).

• Download figure
• Open in new tab

Figure 3.

Sensitivities (dots) and specificities (triangles) with 95% confidence intervals of the antigen rapid test with nasal sampling-RT-PCR reference standard test comparison by week of inclusion, before (left) and after application of a viral load cut-off (right). The red bars indicate the percentage of the Omicron variant according to the national pathogen surveillance, while the numbers indicate the number of participants included in each week.

Stratification of the time trend analysis by reason for testing revealed that these time trends were similar but more pronounced in non-confirmatory testers than in confirmatory testers, although confidence intervals are wide (Figure S5).

Discussion

This large diagnostic accuracy evaluation of three commercially available Ag-RDTs in symptomatic individuals showed overall sensitivities ranging from 70% to 79%, increasing to 77% to 85% when a viral load cut-off was applied, for unsupervised nasal self-sampling during a period in which Omicron was dominant. Emergence of Omicron was associated with a 6 to 13 percentage points decline in overall Ag-RDTs’ sensitivities when combined with nasal self-sampling, although this trend was only significant for Clinitest. The overall sensitivities of the MPBio and Clinitest Ag-RDTs increased from 70% to 83% and 70% to 77%, respectively, when combined with OP-N self-sampling instead of nasal self-sampling only. All above-mentioned sensitivities were substantially higher in confirmatory testers than in those who visited test-sites for other reasons.

Our pre-Omicron studies, and when less than 5% of participants were confirmatory testers, found sensitivities of 72-83% for three different Ag-RDTs when performed by trained professionals, and 78.5% for the SD Biosensor Ag-RDT when performed with unsupervised self-sampling from the nose.^7,18,19 The sensitivities that we found in the first week of the current study, when Delta was still highly dominant, were similar (Flowflex 87%, MPBio 80%, and Clinitest 83%), although the percentage of confirmatory testers in this study was much higher (21%-24%). However, sensitivities declined to 80%, 70%, and 70%, respectively, in the Omicron period. Recent studies from the USA and Italy that evaluated Ag-RDTs when Omicron was dominant, found comparable sensitivities of 74% (128/173 RT-PCR positives) and 82% (126/154 RT-PCR positives), but sampling was performed by professionals and sample sizes were smaller.^11,12

We postulate several potential reasons for the somewhat lower sensitivities with Omicron. First, mutations in the Omicron’s Nucleocapsid (N) protein, the target of Ag-RDTs, could influence binding efficiency of antibodies used in the Ag-RDTs. However, analytical sensitivity based on isolated Omicron and Delta virus generally appeared similar.⁹ Second, the proportion of confirmatory testers, who obviously have a higher a-priori chance of testing positive on the Ag-RDT, may have fluctuated over time and by test-site, although our time trend analyses did not confirm this hypothesis (see below). Third, a larger proportion of individuals over time have experienced a SARS-CoV-2 infection, which may have affected Ag-RDT performance (see below). Fourth, during the study period the participating test-sites and laboratories experienced increasing requests for testing, exceeding the maximum capacity of the Dutch testing infrastructure. As a result, the exposure-testing intervals of participants may have been increased during those weeks, resulting in somewhat lower viral loads at the time of inclusion in the study.

The largest differences in RT-PCR positivity percentages and Ag-RDTs’ performance were between confirmatory testers and individuals who attended the test-sites for other reasons. As expected, RT-PCR positivity percentages were close to 100% in the confirmatory testers and substantially lower (30% to 43%) in the ‘other reasons’ group. This agreed with test positivity percentages observed in national surveillance during the study. Logically, a higher proportion of confirmatory testers (93-95%) than other testers (76-77%) had viral loads above the used viral-load cut-off. Adding oropharyngeal to nasal sampling had a larger benefit in the other reasons group (10-18% increase in sensitivity) than in the confirmatory testers group (<1-4%) because the sensitivities were already very high in the latter group.

We found trends towards lower sensitivities in participants who had a previous SARS-CoV-2 infection (60-67%) compared to those who had not (72-83%), with non-overlapping confidence intervals for Flowflex with nasal self-sampling only. This trend was absent for MPBio with OP-N sampling. These findings should be interpreted with caution due to the larger uncertainty around these subgroup specific accuracy estimates. However, we observed similar trends for the BD-Veritor and SD-Biosensor Ag-RDTs with either professional or self-sampling in two previous studies.^5,7 The lower sensitivities of Ag-RDTs in individuals with a prior infection may be explained by generally lower viral loads, with some individuals potentially carrying viral RNA in the absence of a productive infection (i.e. no viral antigen production). Another explanation might be that individuals with a prior infection have circulating anti-N-protein antibodies²⁰ which might bind to the N protein produced during the new infection, and thereby hampering the binding of monoclonal antibodies against the N-protein in the test device. These effects might be test device-specific given the variability in the performance across the three AG-RDTs.⁵ We also found trends towards slightly higher sensitivities in participants who never received a COVID-19 vaccination for the MPBio and Clinitest Ag-RDTs with nasal self-sampling, but all confidence intervals overlapped with those who were vaccinated 1, 2 or 3 times and no differential impact was observed when combined OP-N sampling was applied. A detailed discussion of other subgroup analysis results can be found in supplementary material 4.

Our study is the only Ag-RDT diagnostic accuracy study thus far that was conducted during the emergence of Omicron, and that compared Ag-RDTs’ performances with nasal self-sampling versus combined OP-N self-sampling. Additional strengths include the large numbers of participants recruited at multiple test-sites, the low percentage of missing values, reference test sampling and Ag-RDT self-testing within a few hours, unsupervised self-testing conform the real-world context of self-testing, blinding of participants for the reference test result, blinding of laboratory staff for the Ag-RDT result, and the use of a viral load cut-off.

Our study also has some limitations. First, the sample size calculation was based on the primary analysis and diagnostic accuracy parameters are by definition less precise for stratified and weekly analyses. Second, we did not determine the virus lineage in individual samples but relied on the national pathogen surveillance data to estimate the weekly Omicron variant prevalence.¹⁶ This surveillance system includes approximately 2,000 random samples from positive samples across the country on a weekly basis. Since regional variations in the Netherlands are very small (data not shown), we are confident that Omicron’s share was over 90% in all test-sites from 12 January 2022 onwards. Third, the viral load cut-off that we used was the cut-off above which 95% of people with a positive RT-PCR test result had a positive virus culture in our similar previous study.² Those experiments were done when the Alpha variant dominated and participants were mostly unvaccinated. However, we believe that this estimate is still more meaningful than using arbitrary Ct value cut-offs of 25 or 30, as is often done.^21,22

We found that Ag-RDT performance with nasal self-sampling declined during the period that Omicron emerged. We also showed that Ag-RDT performance can be improved by adding oropharyngeal to nasal self-sampling. Therefore, after proper evaluation, Ag-RDT manufacturers may consider extending their instructions for use to include combined oropharyngeal and nasal self-sampling. Positive predictive values were high throughout our study, and symptomatic individuals can therefore rely on a positive Ag-RDT result irrespective of SARS-CoV-2 variant dominance or method of self-sampling. Negative predictive values were much lower. In mid-January 2022, the Dutch government advised all individuals with symptoms to do a self-test but advised vulnerable persons and those in close contacts with vulnerable persons to do RT-PCR tests at the PHS. In case of a negative self-test, individuals would be allowed to go to work or school. Our data show that this does reduce but not minimize transmission risks because of the likelihood of false-negative Ag-RDT results. As per national policy, we recommend that persons who test negative by self-test adhere to the general preventive measures such as applying hand hygiene, ensure indoor ventilation, and wearing mouth-nose masks in crowed places. In case of a positive self-test, self-isolation is required but confirmatory testing seems unnecessary in most situations if the infection pressure is high.

Contributors

KGMM initiated the study. ES, RPV, LH, IKV, WvdB, SDP, EL, MH, RM, CW, IV, CRSN-I, SvdH, JAJWK, JHHMvdW, and KGMM designed the study. ES, RPV, CRSN-I, and KGMM coordinated the study. WvdB, SDP, VFZ, LS, and MK were responsible for laboratory analyses and data processing. ES, RPV, and KGMM verified the underlying data. ES performed the statistical analysis and verified the underlying data in close collaboration with RPV and KGMM. ES, RPV, JHHMvdW, and KGMM drafted the first version of the manuscript. All authors critically read the manuscript and provided feedback. All authors approved the submission of the current version of the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Declaration of interest

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Dutch Ministry of Health, Welfare, and Sport for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Ethical approval

Not required because the study was judged by the METC Utrecht to be outside the scope of the Dutch Medical Research Involving Human Subjects Act (protocol No 21-818 /C). All participants signed an informed consent form before any study procedure.

Data sharing

Individual participant data collected during the study will be available, after deidentification of all participants. Data will be available to researchers who provide a methodologically sound proposal to achieve the aims in the approved proposal. Proposals should be directed to the corresponding author to gain access to the data. Data requestors will need to sign a data sharing agreement.

The corresponding author (KGMM, the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned (and, if relevant, registered) have been explained.

Additional information

The study protocol is available upon request by contacting Karel Moons at k.g.m.moons{at}umcutrecht.nl