Abstract
Background During residency, each resident is observed and receives feedback based on their performance. Residency training is demanding, with a few residents struggling in their academic performance. A competency-based residency training program’s success depends on its ability to identify residents with difficulty during their first year of post-graduate education and to provide them with timely intervention and support.
Objective In large training programs such as Family Medicine, identifying residents at risk of failing their certification exams is difficult. We develop a AI system using state-of-the-art technologies in Machine Learning (ML), Deep Learning (DL), Natural Language Processing (NLP) and Explainable AI (XAI) to detect at-risk residents automatically.
Methods We implemented ML, DL and NLP models for the prediction and its performance analysis. The target variable chosen for the prediction was the determination of whether the resident would fail or pass their certification exam. XAI was used to enhance the understanding of the model’s inner workings.
Results In total, there were 1382 data points of residents. The champion model, Support Vector Machine (SVM), achieved an accuracy of 89.05% and an F1 score of 74.54 for the multiclass classification when multimodal (text and tabular) data was used. This model outperformed the models that only used qualitative or quantitative data exclusively.
Conclusion Combining qualitative and quantitative data represents a novel approach and has provided better classification results. This research demonstrates the feasibility of an automated AI system for the early identification of residents at risk of academic struggle.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The project's scope was reviewed by the Office of Research Ethics and Integrity at the University of Ottawa and was determined to fall within Article 2.5 of the TCPS 2 and was therefore deemed exempt from further review.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
The data utilized in this study are not publicly available and will not be shared.
