Comparison of the Sensitivity and Specificity of Commercial Anti-Dengue Virus IgG Tests to Identify Persons Eligible for Dengue Vaccination

Participant enrollment and ethics statement

Early convalescent specimens were retrospectively obtained from the Sentinel Enhanced Dengue and Acute Febrile Illness Surveillance System (SEDSS), as described previously (12, 13). Samples to evaluate test performance for previous dengue infections were retrospectively obtained from the Communities Organized to Prevent Arboviruses (COPA) project in Puerto Rico that conducts annual serosurveys among participants 1-50 years of age. Participants were recruited into COPA as previously described with written informed consent or assent obtained from all study participants according to a protocol approved by Institutional Review Boards at the Centers for Disease Control and Prevention (CDC) and Ponce Medical School Foundation, Inc (PMSF). We used random samples among non-ill children 9-16 years of age collected in 2018 and 2019 (14). Specimens from primary ZIKV infections were obtained from a pediatric cohort in Nicaragua collected in 2018 and 2019 (15). The ZIKV infections were confirmed by real-time RT-PCR during the acute episodes in 2016, and absence of DENV infections was confirmed in paired annual samples tested with a DENV inhibition ELISA (16).

Test selection

A search was performed for commercial tests that detect anti-DENV IgG. A total of 60 tests (46 ELISA and 14 rapid diagnostic tests [RDT]) from 34 manufacturers were identified. This group was narrowed down to 23 tests (11 ELISAs, 12 RDTs) by selecting only tests that reported performance data on their website or in their instructions for use. All but two tests reported greater than 90% specificity. The tests with sensitivity lower than 75% were removed from consideration. This group was further reduced to 7 tests (5 ELISAs, 2 RDTs) since some manufacturers or representatives were not reached after multiple attempts to contact them by both e-mail and telephone (n=9), or if the manufacturer stated that the product was withdrawn from sale (n=5). The 7 tests selected were included in the first phase of the evaluation to determine performance in early convalescent specimens, as these tests have been manufactured for use in disease diagnosis. Based on their performance, 5 of these tests were selected for further analysis. During this evaluation, two versions of a rapid test manufactured by CTK Biotech became available for the detection of IgG antibodies as an indication of prior DENV infection in asymptomatic individuals. CTK Biotech also offers the Alta rapid test reader (RTR-1) as an optional tool to automate and standardize the presence or absence of test line bands. Therefore, 7 tests moved to another round of the evaluation using a panel of samples from patients with confirmed previous dengue infection: the 5 tests with highest accuracy selected from the evaluation with early convalescent specimens and the 2 CTK test modalities. The 4 best performing tests were further evaluated for detection of previous dengue infection in a survey of healthy individuals. The tests used in these evaluations were purchased directly from the manufacturers, stored at the recommended temperatures, and used prior to their expiration dates. The 9 tests evaluated are listed in Table 1.

Focus reduction neutralization test (FRNT)

FRNT assays for DENV-1-4 were performed blinded at the Laboratory of Viral Diseases, National Institute for Allergy, and Infectious Disease (NIAID), National Institutes of Health (NIH) as previously described (17). Viruses used in the assay included: DENV-1 Western Pacific (Nauru/74), DENV-2 NGC Prototype (1944), DENV-3 Sleman/78, and DENV-4 814669 (Dominica/81). Serum specimens were used at dilutions of 1:5, 1:20, and 1:80 with antibody titers defined as the highest serum dilution that resulted in >50% reduction (FRNT50) in the number of immunostained virus foci. A specimen was considered positive for neutralization against the infecting serotype when the FRNT titer was >4 for DENV-1, >15 for DENV-2, >15 for DENV-3, and >4 for DENV-4 (Supplemental Figure S1). Specimens with no neutralizing antibodies against a virus were given a value of 4. We used the FRNT50, considered the gold standard for measuring neutralizing antibodies against dengue virus and other flaviviruses, in combination with the CDC DENV IgG test as our reference standard.

CDC DENV IgG ELISA

The CDC DENV IgG assay was calibrated based on the NIH FRNT50 assay and a modified version of Miagostovich et. al (18). Capture antibody (4G2 Mouse anti-Flavivirus Envelope Protein, Native Antigen Company, Oxford, United Kingdom) was diluted 1:6,000 in 0.1 M carbonate–bicarbonate buffer pH 9.6 and plates were incubated overnight at 4°C. Plates were washed three times in phosphate buffered saline (PBS) and incubated with blocking buffer (PBS pH 7.4/0.05% Tween 20/3% normal goat serum) for 1 hour at 37°C. After removing blocking buffer, a DENV-1-4 antigen mix (DENV-1-4 virus-like particle [VLP] recombinant antigens) titrated between 0.1 ng/µl-0.7 ng/µl for each antigen, Native Antigen Company, Oxford, United Kingdom) diluted in blocking buffer was added to each well and incubated covered for 1 hour at 37°C. Plates were washed three times in PBS. Each plate contained a high positive control, low positive control, high negative control, two low negative controls, and a cutoff control. Serum specimens, positive controls, negative controls, and a cutoff control were diluted 1:100 in a solution of PBS pH 7.4 containing 3% non-fat dry milk/ 0.05% Tween-20 and tested in duplicate. The positive and cutoff controls were created using pooled early convalescent specimens collected 7–30 days post-onset (DPO) of symptoms from RT-PCR confirmed DENV-1–4 cases and diluted in PBS. Plates were washed three times in PBS and a horse radish peroxidase conjugated anti-human IgG diluted in a solution of PBS pH 7.4 containing 3% non-fat dry milk/ 0.05% Tween-20 was added and incubated in the dark for 1 hour uncovered at room temperature. Plates were washed six times with PBS and ABTS substrate added to each well. After a 60-minute room temperature incubation in the dark, plates were read in a microplate spectrophotometer at 405 nm without a blank. Optical density (OD) values and antibody index ratio (AIR) calculations for the controls and samples are needed for test run validation and interpretation of results, accordingly.

AIR = ([control or sample OD / cutoff control OD] *10) Positive: AIR ≥ 15

Equivocal: AIR 10-14.99

Negative: AIR <10

Non-concordant results were repeated twice by CDC DENV IgG ELISA and FRNT50 to confirm results. All specimens except one were concordant in both assays.

ZIKV EDIII IgG ELISA

The ZIKV EDIII IgG ELISA was performed as previously described in Adams et. al. with the modification of using diluted human monoclonal antibody ZKA190 as a cutoff control for inter-plate consistency (19). High-binding microtiter 96-well plates (Greiner Bio-One) were coated with streptavidin diluted to 4 μg/mL in Tris-buffered saline (TBS, pH 7.4), and incubated for 1 hour at 37°C to ensure optimal binding. After incubation, the plate was decanted, washed thrice with wash buffer (TBS containing 0.2% Tween 20). To minimize non-specific binding, the plate was blocked using a blocking solution (3% milk in TBS containing 0.05% Tween 20) for 1 hour at 37°C. Biotinylated EDIII antigen diluted at a concentration of 2 μg/mL in blocking buffer was captured in plates after a 1-hour incubation at 37°C. Plates were washed thrice with wash buffer. After removal of the blocking solution, serum samples at a 1:20 dilution in blocking buffer were added to each well and incubated for 1 hour at 37°C for binding of ZIKV-specific IgG antibodies to the captured EDIII antigen. Following another series of three washes, goat anti-human IgG alkaline phosphatase-conjugated secondary antibody (Sigma) was added at a 1:2,500 dilution and incubated for 1 hour at 37°C. After a final washing, SigmaFast AP substrate was added, and the reaction was developed in the dark for 30 minutes at room temperature. The plate was then read at a 405 nm wavelength setting without a blank. For validation and interpretation of results, optical density (OD) values and antibody index ratio (AIR) calculations were necessary for controls and samples.

AIR = ([control or sample OD / cutoff control OD] *10)

Positive: AIR > 10

Negative: AIR ≤10

Early convalescent specimens

A serum panel was assembled with early convalescent specimens (EC panel, N=44) collected 7– 30 (average = 12) days post-onset (DPO) of symptoms from cases in a convenience series within SEDSS. These cases were confirmed as either DENV or ZIKV positive or negative by RT-PCR on the first specimen collected from each case (DPO 0-5) and whose immune status was determined based on testing a convalescent sample with the CDC DENV IgG or ZIKV IgG assays, as previously described (19). Unexposed specimens (n=8) were negative in the CDC Trioplex RT-PCR, negative for DENV and ZIKV IgG, and IgM negative in the acute and early convalescent specimens for DENV and ZIKV. Specimens from ZIKV RT-PCR positive cases with negative and DENV IgG negative results were named primary ZIKV (n=14). DENV primary specimens (n=12) were DENV RT-PCR-positive and DENV and ZIKV IgG-negative. DENV secondary specimens (n=8) were DENV RT-PCR positive, negative for ZIKV IgG, and positive for DENV IgG.

Previous DENV infection specimens

The following immune status classification criteria were established for samples from a serosurvey of asymptomatic 9-16 Y/O children (COPA) that were available for this study: Specimens were classified as positive for DENV only if they were positive in the DENV FRNT and CDC DENV IgG ELISA. Specimens that neutralized a single DENV serotype and were not positive in the ZIKV EDIII IgG ELISA were classified as monotypic DENV, and specimens that neutralized two or more DENV serotypes and were not positive by ZIKV EDIII IgG ELISA were classified as multitypic DENV. Specimens that did not neutralize any DENV serotype and were ZIKV EDIII IgG ELISA positive were classified as primary ZIKV. Specimens that neutralized any DENV serotype and were positive in the ZIKV EDIII IgG ELISA were classified as multi-flavivirus. Specimens that did not neutralize DENV nor tested ZIKV EDIII IgG ELISA positive, and specimens that were only positive in the CDC DENV IgG ELISA or FRNT50 were classified as unexposed.

We constructed a panel with 44 specimens selected out of from the total COPA serosurvey to assess previous DENV infections (PDI). The 44 samples were classified according to the above-stated immune status classification criteria for asymptomatic cases with the following proportions: unexposed (n=8), primary ZIKV (n=14), monotypic DENV (n=13) and multitypic DENV (n=9). Multi-flavivirus samples were not included in this panel. We then expanded our evaluation to a panel of 400 samples from the COPA serosurvey, which included a blinded selection of 50 specimens from each year of age (9–16), within a total of 750 participants of that age group. These specimens were classified according to the same immune status clarification criteria (Table 2). FRNT50 titers for the 400 samples for each DENV serotype according to their immune status classification can be found in supplementary Figure S1.

Additional evaluation for cross-reactivity with ZIKV specimens

To ensure that the tests with high performance for pre-vaccination screening with Puerto Rico specimens had little cross-reactivity with ZIKV, we tested an additional 22 well-characterized RT-PCR-confirmed past ZIKV infections from a cohort in Nicaragua and an additional 41 specimens from unexposed healthy individuals from Puerto Rico, both obtained from cases of a convenience series. In addition, 12 challenging samples from primary ZIKV positive patients were selected for their high ZIKV neutralizing and binding antibody levels. The specimens were collected at 3-4 months after RT-PCR confirmed infections in 2016-2017.

All serum specimens used in these evaluations were stored frozen after arrival to the CDC Dengue Branch, thawed and prepared in aliquots for virus neutralization, CDC DENV IgG ELISA, or commercial assay testing.

Specimen testing

Specimen testing was conducted contemporaneously with the index tests and reference standards to ensure consistency and reliability in the results. All tests were performed and interpreted according to the manufacturer’s instructions for use and all specimens were tested in index tests and the reference standard. There was no missing data. ELISA equivocal/borderline/weakly reactive results were considered negative for calculations of test performance. Testing was performed by the same laboratory technician for all ELISA assays. Rapid tests were independently interpreted visually by two laboratory technicians who were blinded to specimen classifications. The CTK OnSite Dengue IgG Rapid Tests were also read with the CTK Alta Rapid Test Reader RTR-1 optimized for visual readout with the large panel (PDI, n=400) collected from healthy children 9-16 years of age in Puerto Rico. There were no adverse events associated with performing the index tests or the reference standard. This study was carried out and documented following the Standards for Reporting Diagnostic Accuracy Studies (STARD) guidelines (20). Participant flow and outcomes for the index tests are depicted using STARD flow diagrams (Figures S2-12). This study was conducted and reported in accordance with the Standards for Reporting Diagnostic Accuracy Studies (STARD) guidelines. The flow of participants for the index tests and their results was presented using STARD diagrams.

Sensitivity

Test sensitivity was measured as the evaluated test’s ability to correctly identify specimens that are positive for DENV IgG antibodies through two refence standards: the CDC DENV IgG ELISA and the DENV FRNT50 assays. A specimen was confirmed as DENV IgG positive if it tested positive in both assays serving as reference standards. Sensitivity was calculated as the number of anti-DENV positive specimens correctly detected by the test under evaluation, divided by the total number of anti-DENV positive specimens confirmed by CDC DENV IgG ELISA and classified as DENV immune by FRNT50, multiplied by 100.

Specificity

Test specificity was measured as the evaluated test’s ability to correctly identify specimens that were not exposed to DENV (negative for DENV IgG antibodies) or were exposed to ZIKV only (positive for ZIKV IgG antibodies but negative for DENV IgG antibodies). For this evaluation, we considered non-concordant specimens—those that test positive in either the CDC DENV IgG ELISA or the DENV FRNT50 assay but not in both—as negative for DENV IgG antibodies. Specificity was calculated by adding the number of specimens that the evaluated tests correctly identified as negative (including true negative, equivocal/borderline/weakly reactive and non-concordant specimens) and only ZIKV exposed, divided by the total number of specimens classified as ZIKV primary (based on a positive result in the ZIKV EDIII IgG ELISA and a negative result in the DENV FRNT50) or DENV unexposed (based on negative results in both the CDC DENV IgG ELISA and the FRNT50), multiplied by 100.

Confidence interval

The 95% confidence interval (CI) of the estimated sensitivity and specificity was calculated using the formula: P±1.96√P(1−P)/N, where P is the sensitivity or specificity, and N is the number of specimens tested. If the sensitivity or specificity was 100% it was substituted with 99.9% for P in the equation.

Performance on early convalescent specimens

Seven tests were first evaluated for their capacity to detect anti-DENV IgG and their potential to minimize detection of cross-reactive ZIKV IgG in early convalescent specimens (n=44) (Table 1). All (7/7) tests had at least overall moderate (75%) to high (100%) sensitivity for anti-DENV IgG (Figure 1). The detection of anti-DENV IgG from secondary DENV was universal for all tests; but differences in test sensitivity were observed for primary DENV. Sensitivity for anti-DENV IgG in primary DENV specimens was lower in RDTs in comparison with ELISA tests. The RDTs detected over half of the primary DENV specimens. ELISA tests displayed high sensitivity for anti-DENV IgG in primary specimens, except for the SCIMEDX Dengue IgG serum microwell ELISA, which had a sensitivity comparable to RDTs. Despite the high sensitivity observed in the Focus DENV IgG DxSelect and InBios DENV Detect IgG ELISAs, they were excluded from further evaluation due to the high (50-57 %) reactivity observed in primary ZIKV specimens. Cross-reactivity in the remaining tests was low (7-14%) and warranted further evaluation. No test had false positives in specimens from unexposed individuals.

• Download figure
• Open in new tab

Figure 1. Comparison of anti-DENV IgG commercial tests evaluated for specificity and sensitivity with specimens collected during early convalescence.

Note: Circles represent the number of specimens tested and are not in the same order by column.

Performance to detect previous dengue infections with a small panel of 44 specimens

The 44-sample panel of specimens from non-ill children was used to assess 7 commercial anti-DENV IgG tests for their performance to detect previous dengue infections (PDI-SP, n=44); which included 5 tests that passed the EC panel and 2 IgG tests from CTK Biotech that became commercially available during the course of our evaluation (Table 1). The tests with the highest performance (specificity and sensitivity, respectively) were the Euroimmun DENV-1-4 IgG ELISA (86%, 86%), the Euroimmun DENV NS1 IgG ELISA (95%, 68%), the CTK Onsite Dengue IgG Rapid Test R0065C (95%, 82%) and the CTK Onsite Dengue IgG Rapid Test R0065C 1.0 (95%, 68%) (Figure 2). The sensitivity of Biocan Dengue IgG/IgM Antibody Test (27%), CTK Biotech OnSite Dengue IgG/IgM Combo Rapid (0%), and SCIMEDX Dengue IgG serum microwell ELISA (23%) was overall very low and not suitable for pre-vaccination screening. The test with the highest ZIKV cross-reactivity was the Euroimmun anti-DENV Type 1-4 ELISA IgG ELISA, which uses VLP antigen and had 3/14 primary ZIKV specimens test positive.

• Download figure
• Open in new tab

Figure 2. Comparison of anti-DENV IgG commercial tests evaluated for specificity and sensitivity of previous dengue infection in serum specimens collected from non-ill children (9-16 years of age).

Note: Circles represent the number of specimens tested and are not in the same order by column.

Performance to detect previous dengue virus infection with a community survey of 400 specimens

The full-scale evaluation was done with the two Euroimmun ELISAs and the two CTK OnSite Dengue IgG rapid tests. From the 750 children (age 9-16) in the COPA cohort, 400 specimens were selected for the test evaluation and categorized by immune status using the DENV/ZIKV FRNT and an in-house DENV IgG assay, with a total of 36 monotypic DENV infections, 77 multitypic DENV infections, 56 primary ZIKV infections, 80 multi-flavivirus infections, and 151 unexposed (Table 2 and supplementary table 1). The four tests evaluated displayed a sensitivity near or above the ACIP recommendation (74.1%-91.2%); and high specificity (91.8%-98.6%); but only both versions of the CTK OnSite Dengue IgG rapid test reached the ACIP recommended target of ≥98% specificity (Table 3). The CTK OnSite rapid tests displayed a similar low proportion (∼1.8%) of false positives in the ZIKV primary and unexposed specimens. Of those two tests, only the CTK OnSite Dengue IgG rapid test R0065C with 76.2% sensitivity surpassed the ≥75% performance recommended by the ACIP. The CTK OnSite Dengue IgG Rapid Test R0065C-1.0 with equipment read yielded a sensitivity of 74.1% and nearly achieved the desired performance standards. The visual reads of the CTK OnSite Dengue IgG rapid test obtained by both laboratory technicians were identical for all specimens. Results showed that although the higher sensitivity (89.6% for R0065C, 81.9% for R0065C-1.0) of the rapid tests obtained when laboratory technicians performed the test read compared to the equipment reads, came at the cost of reduced specificity (95.7% for R0065C, 96.6% for R0065C-1.0) (Table 3). The ELISAs achieved 100% specificity in unexposed specimens but displayed some cross-reactivity against ZIKV. Although the Euroimmun anti-DENV NS1 Type 1-4 ELISA displayed a sensitivity (84.5%) high enough for pre-vaccination screening, it fell short on specificity (97.1%). A higher specificity was observed in the Euroimmun ELISA using NS1 than with the ELISA using VLPs as antigen that yielded a 91.8% specificity and sensitivity of 91.2%. (Table 3). Differences in sensitivity between tests were more pronounced in the detection of monotypic DENV specimens. Sensitivity for monotypic DENV was low to moderate (25.0-72.2%) for most tests. Multitypic DENV specimens and those with DENV and ZIKV exposure (multi-flavivirus) were detected with moderate to high sensitivity (≥83%) for all tests. The two CTK OnSite Dengue IgG rapid test versions showed a reduction in sensitivity when read by the manufacturer’s equipment compared to the visual read (Table 3). Because the Alta reader may not be available in all settings, we evaluated the performance of a combination of tests that could be used without the reader. We found that combining the Euroimmun anti-DENV NS1 Type 1-4 ELISA and the CTK OnSite R0065C read visually by the laboratory technician in a two-test algorithm, resulted in a specificity of 100% [98.2-100.0] and a sensitivity of 80.3% [74.0-85.7]; which meets the ACIP recommended performance criteria. Supplemental Figures 2-12 show flow diagrams for all test combinations indicated in Table 3.

Additional evaluation of cross-reactivity

The current and future challenge of pre-vaccination screening arises largely because many dengue endemic countries also experienced ZIKV transmission. Therefore, to ensure dengue test specificity the Euroimmun anti-DENV NS1 Type 1-4 ELISA and CTK OnSite Dengue IgG rapid tests were evaluated with an additional 63 specimens from unexposed healthy individuals from Puerto Rico and well-characterized RT-PCR-confirmed past ZIKV infections from Nicaragua (Table 4).

The highly specific Euroimmun anti-DENV NS1 Type 1-4 ELISA and CTK OnSite Dengue IgG rapid tests were evaluated with 12 samples from primary ZIKV specimens with high ZIKV neutralizing and binding antibody levels. The Euroimmun anti-DENV NS1 Type 1-4 ELISA IgG ELISA yielded 4/12 positives and the CTK OnSite Dengue IgG Rapid Tests both had 1/12 positives (Table 5). Both numbers are higher than expected and suggest that high ZIKV antibody levels could have an impact on test performance.