Cohen’s kappa statistics as a convenient means to identify accurate SARS-CoV-2 rapid antibody tests

Introduction

Now the number of new COVID-19 patients has recently begun to decrease. Now several countries want to investigate the SARS-CoV-2 infection rate as an epidemiological study to grasp the actual situation, which will help to evaluate whether or not the acquisition of herd immunity will be possible or not.

There are lots of rapid antibody tests using lateral flow immunoassays method (LFIA). However, their performance remains unclear in large part. It is necessary to estimate the performance of such rapid antibody tests.

It is difficult to judge whether or not a person is infected with COVID-19 under these circumstances, and more than 35% of such patients show poor symptoms ⁽¹⁾, and the sensitivity of reverse transcription-polymerase chain reaction (RT-PCR) assays are reported to be 70% at most.⁽²⁾ As a result, it is difficult to estimate the agreement between rapid antibody tests and COVID-19 disease in the absence of a reference standard.

We planned to estimate the agreement between rapid antibody tests and RT-PCR using Cohen’s kappa statistics. Kappa (κ) is defined as the difference between the observed and expected agreement expressed as a fraction of the maximum difference and ranges between −1 to 1 ⁽³⁾.

The accepted data are as follows; “A report from the National COVID Scientific Advisory Panel” (England) ⁽⁴⁾, “Application of A Rapid IgM-IgG Combined Antibody Test for SARS-CoV-2” (China) ⁽⁵⁾, and Sagamihara Kyodo Hospital ‘s trial (Japan).

Methods

Agreements between PCR and rapid antibody tests were calculated using the kappa statistics on samples from the English literature, the Chinese literature and our hospital’s patients, who were suspected to have COVID-19 from late April 2020.

We could also accept results using Enzyme-linked immunosorbent assay (ELISA) data in an English report. Kappa statistics was calculated as 1.0 between PCR and ELISA when restricted to samples collected beyond 10 days of symptom-onset. It takes more than a week to produce antibodies in the body. In addition, the PCR reaction becomes negative during the convalescent period.

Judging by the results of this survey, we concluded that samples should be collected beyond 10 days of symptom-onset until the end of the active period as an appropriate period for the agreement between PCR and rapid antibody tests. We regret details regarding the timing of the collected samples from both reports were unknown. For our hospital data, however, we could evaluate the agreement between PCR and rapid antibody test during the appropriate period.

Results

1. In an English report (Table I & Table II)

   Agreement between PCR and ELISA was evaluated using kappa statistics. Kappa statistics was calculated as 0.86±0.03 from the overall data. On the other hand, kappa statistics was 1.00 from the data collected beyond 10 days of symptom-onset. (Table I). Agreement between PCR and antibody tests from 9 devices was evaluated using kappa statistics. The average value was 0.64±0.03. The details are shown in the Table II.

2. In a Chinese report (Table III)

   Agreement between PCR and overall 434 samples of antibody tests were evaluated. Kappa statistics was calculated as 0.75.

3. For our hospital’s data (Japan) (Table IV)

   In the overall data, sensitivity was 0.84, specificity was 0.97, and kappa statistics was calculated as 0.83±0.04. On the other hand, kappa statistics was 0.97±0.02 from the data collected beyond 10 days of symptom-onset.

Sensitivity and specificity were 95.2% and 100% when restricted to these data. About our excluded data

1. PCR(+)IgG(-): 3samples within 10 days of symptom-onset

2. PCR(+)IgG(+): one sample collected at the 8^thday of symptom-onset

3. PCR(-)IgG(+): a patient suffered from an unknown type of pneumonia in March. The patient was in the convalescent period

Discussion

PCR shows a positive finding early in the active period, but it becomes negative in the convalescent period. On the other hand, antibody test shows a negative finding early in the active period because it takes more than a week to produce antibodies in the body.

In an Oxford study, ELISA data show all samples positive beyond 10 days of symptom-onset. Base on this result, samples beyond 10 days in the active period were accepted for assessing agreement in the absence of a reference standard.

Two reports did not show the details of samples. It would be better if samples could be narrowed down to those collected beyond 10 days.

In our hospital’s data, we calculated kappa statistics using the overall data at first, which was 0.83±0.04.

Then we calculated kappa statistics for samples beyond 10 days, which was 0.97±0.02. Sensitivity was 95.2% and specificity was 100%. These were considered to be excellent results.

Our study shows a simple evaluation of antibody tests using kappa statistics between PCR and antibody tests in the appropriate period for the collection of samples.

Regarding limitations associated with this study, our positive data could not be collected as well as expected because SARS-CoV-2 infection rate has recently decreased, and we could not collect as many samples as we had expected.

From our study, however, we believe the proper antibody test kits can be identified using kappa statistics.

CONFLICT OF INTERESTS

The authors declare that there is no conflict of interests.