Abstract
Introduction AI based software, including computer aided detection software for chest radiographs (CXR-CAD), was developed during the pandemic to improve COVID-19 case finding and triage. In high burden TB countries, the use of highly portable CXR and computer aided detection software has been adopted more broadly to improve the screening and triage of individuals for TB, but there is little evidence in these settings regarding COVID-19 CAD performance.
Methods We performed a multicenter, retrospective cross-over study evaluating CXRs from individuals at risk for COVID-19. We evaluated performance of CAD software and radiologists in comparison to COVID-19 laboratory results in 671 individuals evaluated for COVID-19 at sites in Zambia and Malawi between January 2021 and June 2022. All CXRs were interpreted by an expert radiologist and two commercially available COVID-19 CXR-CAD software.
Results Radiologists interpreted CXRs for COVID-19 with a sensitivity of 73% (95% CI: 69%-76%) and specificity of 49% (95% CI: 40%-58%). One CAD software (CAD2) showed performance in diagnosing COVID-19 that was comparable to that of radiologists, (AUC-ROC of 0.70 (95% CI: 0.65-0.75)), while a second (CAD1) showed inferior performance (AUC-ROC of 0.57 (95% CI: 0.52-0.63)). Agreement between CAD software and radiologists was moderate for diagnosing COVID-19, and very good agreement in differentiating normal and abnormal CXRs in this high prevalent population.
Conclusions The study highlights the potential of CXR-CAD as a tool to support effective triage of individuals in Malawi and Zambia during the pandemic, particularly for distinguishing normal from abnormal CXRs. These findings suggest that while current AI-based diagnostics like CXR-CAD show promise, their effectiveness varies significantly. In order to better prepare for future pandemics, there is a need for representative training data to optimize performance in key populations, and ongoing data collection to maintain diagnostic accuracy, especially as new disease strains emerge.
Author Summary During the COVID-19 pandemic, AI-based software was developed to help identify and manage cases, including software that assists in reading chest X-rays (CXR-CAD). This technology has also been used in high tuberculosis (TB) burden countries to screen and manage TB cases. However, there’s limited information on how well these tools work for COVID-19 in these settings. This study examined chest X-rays from people at risk for COVID-19 in Zambia and Malawi to evaluate the performance of CXR-CAD software against expert radiologists and laboratory COVID-19 tests. The research included X-rays from 671 participants, reviewed by two AI software programs and radiologists.
The results showed that radiologists had a sensitivity of 73% and specificity of 49% in detecting COVID-19. One AI software (CAD2) performed similarly to radiologists, while another (CAD1) performed worse. The agreement between the AI software and radiologists varied, but both were good at distinguishing between normal and abnormal X-rays.
The study suggests that while AI tools like CXR-CAD show potential, their effectiveness can vary. To improve these tools for future pandemics, more representative training data and continuous data collection are necessary.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
Yes
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The study received approval through the Clinton Health Access Initiative Institutional Review Board (CHAI IRB), and received in country ethical/IRB approval.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data used in this submission can be accessed. However, all patient medical images are covered under Malawi and Zambia data privacy laws and may not be freely available. De identified CAD Software results and metadata would be made available upon request and anonymized images may be made available pending approval from the Malawi and Zambai MoH.
