Response to Public Reviews
Response to Reviewer #1 (Public Review):
We thank the reviewer for their kind comments and were glad to learn that the reviewer felt the manuscript significantly contributed to body of knowledge on COVID-19.
Response to Reviewer #2 (Public Review):
We thank the reviewer for their dedication to a detailed review, and we greatly appreciated the constructive suggestions given that have helped strengthen the overall manuscript.
We agree with the reviewer that the loss of power for detecting a loss of sensitivity by the ID NOW PCR assay was hindered by the overall low population frequency of COVID-19 disease; this resulted in a low number of positive patients and ultimately led to early study termination as a result. Because of the reviewer’s helpful observation, we have now re-estimated the power of
this study without reference to the observed results, but with consideration for the sample size and proportion of RT-PCR positive tests that were observed when the study was terminated. This new re-estimation suggests that the study retained 80% power to find a difference of 15% or more in sensitivity between ID NOW isothermal PCR and conventional RT-PCR; this analysis also
demonstrated over 95% power to find a difference in specificity of more than 5%. Indeed, the significant drop in population prevalence that led to a loss of power for detecting loss of sensitivity expectedly resulted in an increase in power for detecting loss of specificity. We have expanded the Methodssection of the paper to better expose these issues, and we have expanded
our statement of strengths and limitations in the Discussion for the same reason.
The Methods section of the manuscript now reads as follows:
"The original study design called for enrolling 2000 symptomatic and 500 asymptomatic subjects, which would have provided, in the symptomatic population, power of 80 % for finding a difference (at α = 0.05) of 5% in the sensitivity of ID NOW compared with the RTPCR reference standard; inclusion of at least 1350 negative patients would have provided 95% power (at α =0.025) for finding a 5% difference in specificity. The study design assumed a population prevalence of 10%, and the study was terminated early when the population prevalence dropped to such a low level as to make the study unaffordable. We have re-estimated the power of this study without reference to the observed results
but considering the sample size and proportion of RT-PCR positive tests that were observed when the study was terminated. This re-estimation suggests that the study retained 80% power to find a difference of 15% or more in sensitivity between ID NOW and RTPCR, and well over 95% power to find a difference in specificity of more than 5%. Indeed, the significant drop in population prevalence that led to a loss of power for detecting loss of sensitivity resulted, as expected (Bujang and Adnan, 2016), in an increase in power for detecting loss of specificity.
The revised section of the Discussion regarding strengths and weaknesses now reads as follows:
"Our clinical study also suffered a significant loss of power to assess ID NOW sensitivity as a result of the low number of positive results, and the reduction of sample size caused by the decision to terminate the study as a result. The meta-analysis is also limited by the small number of studies meeting inclusion criteria, and the fact that positive cases are heavily concentrated in only a single study. Strengths of the clinical study include pretrial power analysis with sample size estimation, precise adherence to the ID NOW specimen acquisition protocol, and extremely high power for assessing assay specificity. Taken together with the focus on initial diagnosis of disease in the studies included in the meta-analysis, we believe the combination of trial and meta-analysis provides useful
information for clinicians for whom point-of-care testing is helpful."
We thank the reviewer for noting the unique findings from the current cohort study in comparison to existing literature. The current cohort study was done with meticulous care to identify apparent “false positives” returned by ID NOW PCR assay. This is also reflected in some of the high-quality studies available in the literature. In the supplementary data, we have provided confusion matrices for all the studies included in the meta-analysis. We have identified four such cases out of 1,942 total ID NOW tests. Cell sizes of 0 (from our study) and 4 are two small to allow use of Chi-squared for assessment of heterogeneity, and unfortunately the total
number of tests is too large to allow computation of Fisher’s exact test; however, with such small numbers it is reasonable to treat them as samples drawn from Poisson distributions. The confidence interval around a Poisson estimate of 4 is 1.08987 ≤ μ ≤ 10.24159, and that around a Poisson estimate of 0 is 0.00000 ≤ μ ≤ 3.68888. The overlap is such that the estimates, 0 and 4,
are consistent with having sampled the same distribution. While this does not allow us to conclude that no difference exists between our results and those of the other studies, it does not provide any evidence that there is a difference from the current cohort study and those previously published.
