Abstract
Immigrants and refugees seeking admission to the United States must first undergo an overseas medical exam, overseen by the US Centers for Disease Control and Prevention (CDC), during which all persons ≥15 years old receive a chest x-ray to look for signs of tuberculosis. Although individual screening sites often implement quality control (QC) programs to ensure radiographs are interpreted correctly, the CDC does not currently have a method for conducting similar QC reviews at scale.
We obtained digitized chest radiographs collected as part of the overseas immigration medical exam. Using radiographs from applicants 15 years old and older, we trained deep learning models to perform three tasks: identifying abnormal radiographs; identifying abnormal radiographs suggestive of tuberculosis; and identifying the specific findings (e.g., cavities or infiltrates) in abnormal radiographs. We then evaluated the models on both internal and external testing datasets, focusing on two classes of performance metrics: individual-level metrics, like sensitivity and specificity, and sample-level metrics, like accuracy in predicting the prevalence of abnormal radiographs.
A total of 152,012 images (one image per applicant; mean applicant age 39 years) were used for model training. On our internal test dataset, our models performed well both in identifying abnormalities suggestive of TB (area under the curve [AUC] of 0.97; 95% confidence interval [CI]: 0.95, 0.98) and in estimating sample-level counts of the same (−2% absolute percentage error; 95% CIC: −8%, 6%). On the external test datasets, our models performed similarly well in identifying both generic abnormalities (AUCs ranging from 0.89 to 0.92) and those suggestive of TB (AUCs from 0.94 to 0.99). This performance was consistent across metrics, including those based on thresholded class predictions, like sensitivity, specificity, and F1 score.
Strong performance relative to high-quality radiological reference standards across a variety of datasets suggests our models may make reliable tools for supporting chest radiography QC activities at CDC.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
The study did not receive any funding.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This project was proposed, reviewed, and approved in accordance with Centers for Disease Control and Prevention institutional review policies and procedures. Because it received a non-research determination, review by an institutional review board was not required.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Disclaimer: The findings and conclusions in this manuscript are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention (CDC).
Data Availability
Due to the sensitive nature of the populations under study, and to ensure applicant privacy, neither trained model weights nor raw images will be made publicly available.
Abbreviations
- AI
- artificial intelligence
- AP
- anterior-posterior
- AUC
- area under the ROC curve
- CDC
- Centers for Disease Control and Prevention
- M
- CNN
- convolutional neural network
- DGMH
- Division of Global Migration Health
- HaMLET
- Harnessing Machine Learning to Eliminate Tuberculosis
- IOM
- International Organization for Migration
- IRHB
- Immigrant and Refugee Health Branch
- MiMOSA
- Migrant Management Operational System Application
- MTB
- Mycobacterium tuberculosis
- NLP
- natural language processing
- NAAT
- nucleic acid amplification testing
- OCR
- optical character recognition
- PII
- personally-identifiable information
- PPV
- positive predictive value
- QC
- quality control
- ROC
- receiver operating characteristic
- TPP
- target product profile
- WHO
- World Health Organization
