Abstract
Background Crohn’s Disease (CD) patient heterogeneity in clinical practice is captured by the Montreal Classification. While the underlying concepts, disease behavior and age at diagnosis, are relevant outcomes and covariates in studies from real-world data, extracting this clinical information through manual chart review is labor-intensive and with limited scalability.
Methods We developed and evaluated automated phenotyping algorithms to extract disease behavior and age at diagnosis from clinical narrative texts, using a rule-based approach based on the spaCy framework, and an approach based on zero-shot inference. The underlying data included 49,572 clinical notes and 2,204 radiology reports from 584 CD patients of the Mount Sinai Crohn’s and Colitis Registry. A test set of 200 clinical texts per classification category was labeled at sentence-level, in addition to patient-level ground truth data. The algorithms were evaluated based on their recall, precision, specificity values, and F1-scores.
Results For the labeled dataset, an overall Cohen’s kappa inter-annotator agreement of 0.84 was achieved. The rule-based approach yielded high recall and precision values (0.75 - 1.00) on a note level for the behavioral disease phenotype using clinical notes, with slightly reduced performance using radiology reports. For age at diagnosis, recall and precision values of 0.81 and 0.88 were achieved on note-level, respectively. For both categories, the performance on patient-compared to note-level was reduced, potentially due to the accumulation of false positives and limitations in the data availability.
Conclusion Based on our newly annotated dataset, we demonstrated the feasibility of automatically extracting disease behavior and age at diagnosis from clinical text. The resulting labels may facilitate extensive cohort analyses based on electronic health records, and support chart review processes in the future.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work is supported in part through the use of the research platform AI-Ready Mount Sinai (AIR.MS), and through the MSDW resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai. This work was funded by the Joachim-Herz Foundation (to SI), and the Hasso Plattner Foundation (HPF).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The study was approved by the Program of Protection for Human Subjects at the Icahn School of Medicine at Mount Sinai.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
The data are not publicly available as it contains protected health information.
