Abstract
Background Antigen lateral flow devices (LFDs) have been widely used to control SARS-CoV-2. Changes in LFD sensitivity and detection of infectious individuals during the pandemic with successive variants, vaccination, and changes in LFD use are incompletely understood.
Methods Paired LFD and PCR tests were collected from asymptomatic and symptomatic participants, across multiple settings in the UK between 04-November-2020 and 21-March-2022. Multivariable logistic regression was used to analyse LFD sensitivity and specificity, adjusting for viral load, LFD manufacturer, setting, age, sex, assistance, symptoms, vaccination, and variant. National contact tracing data were used to estimate the proportion of transmitting index cases (with ≥1 PCR/LFD-positive contact) potentially detectable by LFDs over time, accounting for viral load, variant, and symptom status.
Findings 4131/75,382 (5.5%) participants were PCR-positive. Sensitivity vs. PCR was 63.2% (95%CI 61.7-64.6%) and specificity 99.71% (99.66-99.74%). Increased viral load was independently associated with being LFD-positive. There was no evidence LFD sensitivity differed between Delta vs. Alpha/pre-Alpha infections, but Omicron infections were more likely to be LFD positive. Sensitivity was higher in symptomatic participants, 68.7% (66.9-70.4%) than in asymptomatic participants, 52.8% (50.1-55.4%). 79.4% (68.6-81.3%) of index cases resulting in probable onward transmission with were estimated to have been detectable using LFDs, this proportion was relatively stable over time/variants, but lower in asymptomatic vs. symptomatic cases.
Interpretation LFDs remained able to detect most SARS-CoV-2 infections throughout vaccine roll-out and different variants. LFDs can potentially detect most infections that transmit to others and reduce risks. However, performance is lower in asymptomatic compared to symptomatic individuals.
Funding UK Government.
Evidence before this study Lateral flow devices (LFDs; i.e. rapid antigen detection devices) have been widely used for SARS-CoV-2 testing. However, due to their imperfect sensitivity when compared to PCR and a lack of a widely available gold standard proxy for infectiousness, the performance and use of LFDs has been a source of debate. We conducted a literature review in PubMed and bioRxiv/medRxiv for all studies examining the performance of lateral flow devices between 01 January 2020 and 31 October 2022. We used the search terms ‘SARS-CoV-2’/’COVID-19’ and ‘antigen’/’lateral flow test’/’lateral flow device’. Multiple studies have examined the sensitivity and specificity of LFDs, including several systematic reviews. However, the majority of the studies are based on pre-Alpha infections. Large studies examining the test accuracy for different variants, including Delta and Omicron, and following vaccination are limited.
Added value of this study In this large national LFD evaluation programme, we compared the performance of three different LFDs relative to PCR in various settings. Compared to PCR testing, sensitivity was 63.2% (95%CI 61.7-64.6%) overall, and 71.6% (95%CI 69.8-73.4%) in unselected communitybased testing. Specificity was 99.71% (99.66-99.74%). LFDs were more likely to be positive as viral loads increased. LFD sensitivity was similar during Alpha/pre-Alpha and Delta periods but increased during the Omicron period. There was no association between sensitivity and vaccination status. Sensitivity was higher in symptomatic participants, 68.7% (66.9-70.4%) than in asymptomatic participants, 52.8% (50.1-55.4%). Using national contact tracing data, we estimated that 79.4% (68.6-81.3%) of index cases resulting in probable onward transmission (i.e. with ≥1 PCR/LFD-positive contact) were detectable using LFDs. Symptomatic index cases were more likely to be detected than asymptomatic index cases due to higher viral loads and better LFD performance at a given viral load. The proportion of index cases detected remained relatively stable over time and with successive variants, with a slight increase in the proportion of asymptomatic index cases detected during Omicron.
Implications of all the available evidence Our data show that LFDs detect most SARS-CoV-2 infections, with findings broadly similar to those summarised in previous meta-analyses. We show that LFD performance has been relatively consistent throughout different variant-dominant phases of the pandemic and following the roll-out of vaccination. LFDs can detect most infections that transmit to others and can therefore be used as part of a risk reduction strategy. However, performance is lower in asymptomatic compared to symptomatic individuals and this needs to be considered when designing testing programmes.
Introduction
Early detection of symptomatic and asymptomatic SARS-CoV-2 infections has been a key control measure during the COVID-19 pandemic. Rapid point-of-care antigen detection lateral flow devices (LFDs) have been widely in testing in the United Kingdom (UK),[1,2] including, at different times, for population-wide asymptomatic screening, and screening in specific groups such as healthcare workers, school-age children, and populations with increased incidence. LFDs have also been used to allow contacts of infected cases to continue to work or attend education, as well as prior to travel, attending events, and visiting residential care facilities.
However, LFDs have generated considerable scientific and policy debate.[3] They have imperfect sensitivity relative to PCR testing: ranging from <50% to >80%.[4–6] Sensitivity is greater in symptomatic vs. asymptomatic infection, e.g. 72% vs. 58%.[4] This had led to concerns that LFDs may miss some infections and paradoxically increase transmission if individuals testing falsely-negative reduce transmission precautions.[7] Conversely, LFDs need not detect and prevent all transmission to still have an effect at a population level; particularly where reproduction numbers are marginally over 1, an imperfect intervention may still be sufficient to control an outbreak.[8–11]
An additional concern is the lack of a widely-available proxy for an individual being infectious, as PCR-positivity may persist for days to weeks following the end of symptoms and/or infectiousness. It is therefore difficult to directly assess the sensitivity of LFDs in those who are infectious, which would be a better measure of their performance as a control intervention than sensitivity relative to PCR testing.[12] The likelihood of onward transmission is also related to index case viral load, with lower measured PCR cycle threshold (Ct) values (a proxy for higher viral loads in the tested individual) being associated with more secondary cases.[13,14]
LFDs are more sensitive as viral load increases,[4,14] such that performance may be better understood by a curve reporting sensitivity at various viral loads, rather than simply a single figure that also depends on the distribution of viral loads in the population studied. By combining index case PCR viral loads, contact tracing data, and LFD-viral load performance curves we have previously shown that LFDs would be expected to have detected most infections leading to onward transmission in the Alpha period in the UK.[14]
Here we present data on the performance of LFDs relative to PCR across multiple evaluations conducted in a range of settings during phases of the pandemic where the Alpha, Delta and Omicron variants dominated, to evaluate if performance has remained stable over time, including with any changes in population-wide proficiency and use of the tests, following vaccinations and with viral evolution. We combine our findings with contact tracing data to also evaluate the proportion of infectious cases detected by LFDs over time.
Methods
LFD evaluations: participants and samples
Prospective collection of paired LFD results and samples for PCR testing was undertaken to evaluate the performance of LFDs and their deployment, and for service quality assurance by NHS Test & Trace (now part of the UK Health Security Agency, UKHSA). Participants were asked to take a second test as well as their standard test specifically for the purposes of evaluating the LFDs and the testing programme. Prior to wider national deployment, the performance of Innova LFDs was evaluated in two field studies, referred to as “predeployment testing”. We analysed data collected from the start of the evaluation programme until the end of provision of free testing for the general public in March-2022.
Paired PCR and LFD testing was undertaken in several settings. Pre-deployment data were obtained from symptomatic testing sites, with a mixture of assisted and self-taken swabs. Subsequent settings included testing offered to asymptomatic individuals in the community (from the general public via testing sites and home testing; city-wide testing in Liverpool during a period of increased incidence[15]); targeted community testing (disproportionately impacted and under-served groups); schools (predominantly secondary); private and public sector workplaces; universities; and healthcare staff. Paired testing was also undertaken at local and regional symptomatic testing sites.
Participants were provided with instructions on how to perform testing. Two separate combined throat and/or nose swabs were obtained for testing (see Supplement). Swabs were self-taken. In assisted testing the LFD result was interpreted by a trained individual. In self-testing, the user (or relevant person on behalf of the user such as a parent/guardian/carer) self-swabbed and interpreted the test themselves without assistance. Evaluations were carried out in live testing services, therefore standard testing was always prioritised over the supplementary test, meaning there was no randomisation of swabbing order during sample collection. If standard testing was LFD, participants were asked to put the LFD to one side to develop, while providing the PCR sample (so that they were unaware of the result of either test at sampling). Participants or operatives (where testing was assisted) were asked to interpret LFD results as positive, negative, or void. Participants were unaware of PCR results at the time of LFD testing. Similarly, laboratories undertaking PCR testing were unaware of LFD results.
LFD evaluation: sample assays
LFDs evaluated were the (i) Innova SARS-CoV-2 Lateral Flow Antigen Test (Innova in original packaging or repacked with individual buffer containers, also known as Biotime); (ii) Orient Gene COVID-19 Ag Rapid Test Cassette LFD antigen tests (Orient Gene); and (iii) Acon Flowflex SARS-CoV-2 antigen rapid test (Self-Testing) kit (Acon). PCR testing was undertaken by routine laboratories within the NHS Test and Trace laboratory network, and was performed predominantly using the Thermo Fisher SARS-CoV-2 TaqPath assay.
Contact tracing data
National contact tracing data from England was obtained from NHS Test & Trace, as described previously.[13,14] Index cases were notified to the service following a positive community or healthcare-based SARS-CoV-2 PCR or LFD result. All index cases with a diagnostic PCR test performed using the Thermo Fisher SARS-CoV-2 TaqPath assay at three national testing “Lighthouse” laboratories in Milton Keynes, Alderley Park and Glasgow were eligible for inclusion.
Contacts (persons living in the same household, or in face-to-face distance from an index patient, within <1 m for ≥1 minute or within <2 m for ≥15 minutes) were eligible for inclusion in the study, provided they were only named by a single index patient in the 14 days either side of the index patient’s positive test. Contacts with any positive PCR or LFD result in the 1 to 10 days after the index patient’s positive test were considered to represent plausible transmission events. The 1 to 10 day period was chosen to enrich for contacts for whom the index patient was the most likely source of any infection, as previously described.[13,14]
Statistical analysis
For LFD performance evaluations the infecting variant in PCR-positive infections was assigned based on sequencing/genotyping where available and if not, based on the dominant circulating variant (see Supplement). Real-time quantitative PCR cycle threshold (Ct) values were used to estimate SARS-CoV-2 viral loads in swab fluids in copies/mL using calibrant samples (Table S1).
PCR-positive samples were used for analyses of LFD sensitivity. Univariable and multivariable logistic regression was used to model the relationship between LFD positivity and log10 viral load and other covariates. Covariates included LFD device, study setting, assisted vs. self-testing, self-reported symptom status (symptomatic, i.e. fever, cough or anosmia/ageusia, otherwise asymptomatic), vaccination status by number of doses (0, 1, 2 or more doses) and viral variant (Alpha [B.1.1.7] / pre-Alpha [B.1.177], Delta [B.1.617.2], Omicron [BA.1 and BA.2]; or Other / Unknown). PCR-negative samples were used to analyse LFD specificity using univariable and multivariable logistic regression and the same covariates.
We used contact testing data and logistic regression to estimate the relationship between index case symptom status and PCR Ct values/viral loads, and positive results in PCR/LFD-tested contacts. We followed a similar approach to previous analyses[13,14] adjusting for multiple index case and contact factors (see Supplement). We used index case-contact pairs plausibly related by transmission to estimate the proportion of infectious index cases potentially detected by LFDs (see Supplement).
Ethics
Public Health England’s Research Ethics (PHEREG) provided approval for the studies as Service Evaluation and Ongoing Evaluation. This was reviewed and approved under REGG R and D 438 (see Supplement for further details).
Role of the funding source
LFD evaluations were commissioned by the UK Health Security Agency (UKHSA) and the Department of Health and Social Care (DHSC). Data were analysed by an independent team at the University of Oxford. All authors, including those from the UKHSA and DHSC, contributed to writing the report.
Results
83,280 paired LFD and PCR tests were performed. 7898 were excluded: 23 were performed with an LFD manufacturer other than the three evaluated, 6389 were tested with endpoint PCR which does not produce comparable estimates of viral load [16] or an unknown PCR type rather than real-time PCR, 46 where the PCR testing was performed at laboratories not participating in the study, 1272 with a void PCR result, 21 participants withdrew from the study, and 147 had void LFD results. This left 75,382 pairs of samples available for analysis performed between 04-November-2020 and 21-March-2022, including 4131 (5.5%) PCR-positive samples (Figure 1). Tests were performed across multiple settings: pre-deployment evaluation (PCR-positive/n 1123/6759, 16.6%), community-based testing (2381/32,266, 7.4%), healthcare workers (28/1587, 1.8%), schools (89/12,397, 0.7%), workplaces (7/5700, 0.1%), universities (38/6456, 0.6%), and in specifically targeted unrepresented groups (465/10,217, 4.6%) (Figure S1).
Lateral flow device sensitivity
The overall sensitivity of LFDs relative to PCR-testing was 63.2% (2609/4131, 95%CI 61.7-64.6%). Sensitivity was 71.6% (95%CI 69.8-73.4%) in unselected community-based testing, which was higher than in pre-deployment testing, 52.8% (49.8-55.8%) (Table 1). Unadjusted sensitivity was 55.7% (53.1-58.2%), 64.0% (61.6-66.4%), and 73.0% (70.2-75.6%) during the Alpha/Pre-alpha, Delta and Omicron periods respectively. Unadjusted sensitivity was higher following vaccination, i.e., 67.6% (63.3-71.7%), and 69.7% (67.3-72.0%), after 1, and 2 or more doses respectively, compared to 57.3% (55.1-59.4%) in unvaccinated participants. Sensitivity was higher in symptomatic participants, 68.7% (66.9-70.4%) than in asymptomatic participants, 52.8% (50.1-55.4%).
After adjustment for all other factors, increased viral load was independently associated with a positive LFD result (adjusted odds ratio, aOR=2.85 [95%CI 2.66-3.06] per log10 copies/mL higher), but there was no evidence of a difference in sensitivity between the Innova, Acon, and Orient Gene LFDs (Figure 2, Figure S2). Using the Innova LFD as an example, predicted sensitivity (95%CI) for detecting Omicron infection in community-based symptomatic testing was 28.9% (20.8-38.7%), 77.6% (69.6-83.9%), and 96.7% (95.0-97.9%) at viral loads of 103, 105, and 107 copies/mL respectively. Corresponding values for asymptomatic testing were 19.9% (13.6-28.2%), 67.9% (57.9-76.5%), and 94.7% (91.9-96.6%) respectively.
Compared to community-based testing, sensitivity was independently lower in pre-deployment testing (aOR=0.51 [0.36-0.73]), and in targeted groups (aOR=0.51 [0.33-0.78]). Symptomatic participants were independently more likely to test positive (aOR=1.63 [1.30-2.04]), but there was no evidence of a difference in LFD sensitivity with and without assistance performing the test. After adjusting for other factors there was no evidence that vaccination status, age or sex were associated with LFD results. Similarly, there was no evidence that LFD sensitivity differed with Delta infections compared to Alpha/pre-Alpha infections, but Omicron infections were more likely to be LFD positive than Alpha/pre-Alpha infections (aOR=1.63 [1.02-2.59]).
Lateral flow device specificity
The overall specificity of LFDs was 99.71% (71,041/71251, 95%CI 99.66-99.74%) and was ≥99.4% across all study settings (Table 2). In a multivariable model, compared to the Innova LFD, Acon devices were associated with 2-times more false-positive results (aOR=2.00 [95%CI 1.21-3.29]), with moderate evidence Orient Gene devices were associated with fewer false-positive results (aOR=0.49 [0.23-1.04]). Compared to community-based testing there were 67% fewer false-positive results in schools (aOR=0.33 [0.16-0.68]) and likely also fewer in universities (aOR=0.34 [0.10-1.16]), whereas false-positive results were 2-times more common in pre-deployment testing (aOR=2.09 [1.15-3.82]). Self-performed tests resulted in >2.5-times more false-positives than assisted tests (aOR=2.61 [1.46-3.13]). False-positive results were 2-times more common in symptomatic compared to asymptomatic participants (aOR=2.15 [1.48-3.13]). There was no evidence of a difference in false-positive results by vaccination status, age or sex. False-positive tests were more common during the Delta (aOR=2.68 [1.43-5.00]) and Omicron (aOR=5.47 [2.54-11.8]) periods compared to the Alpha/Pre-Alpha (B.1.1.177) period.
Proportion of infectious individuals detected by lateral flow devices
Between 01-January-2021 and 11-January-2022 6,263,786 contacts of PCR-positive SARS-CoV-2 index cases were identified, 1,173,643 (18.7%) underwent a PCR or reported LFD in the 1 to 10 days following the index case’s PCR test, 377,151 (6.0%) tested positive. Contacts were linked to 347,374 unique index cases, with 322,416 (92.8%) index cases linked to a single positive contact, 21,321 (6.1%) to two positive contacts and 3637 (1.0%) to ≥3 (Figure S3).
As described previously,[13] tested contacts of index cases with higher viral loads (lower Ct values) were more likely to test positive. Contacts of symptomatic index cases were more likely to test positive than those of asymptomatic index cases, even at a given viral load (Figure 3). Additionally, viral loads in asymptomatic index cases were lower than those in symptomatic index cases throughout the Alpha, Delta and Omicron periods (Figure S4).
79.4% (95%CI 68.6-81.3%) of index cases with ≥1 PCR/LFD-positive contact were estimated to have been detectable by an LFD. Similar detection rates were seen in index cases with 1, 2 or ≥3 PCR/LFD-positive contacts, 79.4% (68.5-81.2%), 80.2% (69.1-81.7%), 79.6% (68.1-81.2%), respectively. Amongst index cases with ≥1 PCR/LFD-positive contact, 20,712 were recorded as asymptomatic and 326,662 as symptomatic; the proportion of index cases estimated as detectable by LFD was higher with symptoms (80.8% [95%CI 69.3-82.1%]) than without (59.1% [55.2-69.0%]).
The estimated proportion of index cases detected by LFD was similar over time and across the Alpha, Delta, and Omicron periods for symptomatic cases (Figure 4, Table 3). For asymptomatic index cases, the estimated percentage detected by LFDs was consistently lower than for symptomatic index cases, but within asymptomatic index cases higher during the Omicron period compared to the Alpha and Delta periods. This reflects the increase in measured sensitivity of LFDs during the Omicron period rather than a change in the distribution of viral loads (Figure S4).
Discussion
In a national LFD evaluation program, the sensitivity of LFDs compared with PCR was 71.6% (95%CI 69.8-73.4%) in 2381 paired PCR-positive samples obtained in routine community-based testing. There was no evidence that LFD sensitivity or specificity was independently associated with vaccination status, age or sex. Performance was similar during the Alpha/Pre-Alpha and Delta periods, with an increase in sensitivity during the Omicron period. This potentially reflects either intrinsic changes in the SARS-CoV-2 virus or changes in testing proficiency and behaviour around testing over time but is in contrast to other reports of lower sensitivity with Omicron,[17,18] while others have reported similar performance with Omicron vs. Delta.[19] Conversely, although specificity remained >99%, it declined over time with successive variants.
Symptomatic individuals had higher viral loads in the LFD evaluations. However, even having adjusted for this, the presence of fever, cough or anosmia/ageusia were associated with increased LFD sensitivity at a given viral load compared to without these cardinal symptoms. This may reflect different ratios of antigen to RNA with different symptom statuses and day of infection. With a one-off screen asymptomatic individuals may be detected later in their infections where residual RNA may be present with less antigen compared to earlier in infection, when most symptomatic cases were tested. Serial testing by PCR and quantitative antigen tests could help investigate this further.
There was no evidence of a difference in overall sensitivity between the different LFD manufacturers tested, potentially reflecting limited power as most paired tests were performed with the Innova LFD. However, when considering sensitivity by viral load, there was some evidence that the Acon device was more sensitive at lower viral loads, but this was also the device with the lowest specificity, likely reflecting a trade off in how devices are calibrated.
LFD sensitivity increased with viral load, such that those with higher viral loads, i.e. those who were more infectious, were more likely to detected by LFDs.[20–23] Amongst case-contact pairs where transmission was plausible, the percentage of index cases detected by LFDs was 79.4%, similar to previous estimates with the same LFD device.[14] This is higher than the overall sensitivity of LFDs at 71.6% in community testing. However, we show that the potential for LFDs to detect infectious asymptomatic index cases was lower than for symptomatic index cases, at 59.1% vs. 80.8%. This is the result of a combination of reduced sensitivity at a given viral load and reduced viral loads in asymptomatic infections in the contact testing data. However, asymptomatic individuals are less infectious than symptomatic individuals at a given viral load, and the lower viral loads in asymptomatic individuals are also associated with less transmission.[13] It is also worth noting that asymptomatic index cases are likely to be under ascertained in national contact tracing data, especially during periods when asymptomatic screening was uncommon.
Taken together our findings provide support for the use of LFDs as a mechanism to detect potentially infectious individuals and reduce transmission, particularly as these tests perform best in those most likely to be infectious, i.e., those who are symptomatic and/or with high viral loads. However, we show, alongside other studies, that the performance of these tests in asymptomatic individuals may be lower than is generally understood.[4,22,24] LFDs have been deployed in several settings including population-wide mass testing of asymptomatic individuals. In this setting a combination of population prevalence, lower viral loads in asymptomatic individuals, and lower onward transmission at a given viral load compared to symptomatic cases mean that even with perfect LFD sensitivity likely several thousand asymptomatic people need to be tested to detect one case who would otherwise go on to transmit (Table S2). As the sensitivity of LFDs falls, more individuals need to be tested to detect one potentially transmitting case. If false-negative results do not change behaviour, then reduced LFD sensitivity acts simply to make testing programmes less efficient and more costly per transmission averted, but the use of LFD is still potentially effective at reducing the spread of infections. Hence, where people plan to participate in activities anyway then LFDs may act to reduce risk, even if imperfectly. Another setting where risk reduction may be important is where the consequences of transmission are high and those tested are likely to maintain transmission precautions with a negative result, e.g. in healthcare or social care. However, some caution is required if LFDs are used by individuals to relax adhering to transmission precautions.[25,26] Here it is possible that false-negative results with a poorly sensitive test could lead to additional transmission. This possibility needs to be considered when assessing the implementation and messaging around potential asymptomatic screening programs.
This study has several limitations regarding the evaluation of LFD performance. As testing was undertaken in real-world settings, the primary diagnostic test was performed first, rather than randomising the order of swabs taken for LFDs and PCR. A range of different PCR tests were used as a reference standard, however calibrants were used to convert Ct values across assays to estimated viral loads in common units. The calibration approach adopted using a known reference standard was deployable across all test sites, but more accurate approaches for quantifying viral load including droplet digital PCR exist. It should also be noted that viral loads estimated in the samples only approximate viral loads in the participants given respiratory samples are based on swabs rather than direct sampling of a body fluid. Although we show that asymptomatic individuals were less likely to be detected by LFDs, in part this may reflect the performance of a single test which may have captured some infections several days after they started. It is possible that regular asymptomatic testing would have improved performance, as this may catch incident infections earlier in their course. We also did not make use of digital LFD readers which may increase accuracy and have since been widely deployed in the UK.[27,28]
The transmission analysis also has limitations. The case-contact pairs identified rely on both index case and contacts participating in SARS-CoV-2 testing, with several demographic, socioeconomic, and behavioural factors potentially affecting test-seeking. One particular concern is that during the period of the study PCR testing was provided for symptomatic individuals or following a positive LFD test. However, some asymptomatic individuals also sought PCR testing for other reasons, e.g. following contact events. It is therefore possible that case-contact pairs involving an asymptomatic case are enriched for pairs with contact with a third party who is the true source of both infections. In this case the properties of the asymptomatic cases are not what determined transmission.
In summary, LFDs have remained able to detect most SARS-CoV-2 infections throughout the roll-out of vaccination and with several different viral variants. Although on-going monitoring of performance with new variants is required while tests are used, it is reassuring that LFDs are probably likely to remain able to detect future variants. LFDs potentially detect most infections that have the potential to transmit to others, however performance is lower in asymptomatic compared to symptomatic individuals and this needs to be considered when designing testing programs.
Data Availability
Applications to use the data in this study can be made to NHS Digital's Data Access Request Service, please see https://digital.nhs.uk/services/data-access-request-service-dars for more details.
Data availability
Applications to use the data in this study can be made to NHS Digital’s Data Access Request Service, please see https://digital.nhs.uk/services/data-access-request-service-dars for more details.
Transparency declaration
DWE has received lecture fees from Gilead outside the submitted work. No other author has a conflict of interest to declare.
Funding
Supported by the UK Government Department of Health and Social Care; the National Institute for Health Research (NIHR) Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance, University of Oxford, in partnership with Public Health England (NIHR200915); and the University of Oxford NIHR Biomedical Research Centre. DWE is a Robertson Foundation Fellow.