2 Abstract
Importance Identifying opportunities for improvement (OFI), errors in care with adverse outcomes, through mortality and morbidity conferences is essential for improving trauma quality. To screen patients for such conferences, trauma quality improvement programs rely on labor-intensive human reviews and audit filters that exhibit high false positive rates.
Objective This study was conducted to develop machine learning models that predicts OFI in trauma care and compare the performances of these models to those of commonly used audit filters.
Design In this registry-based cohort study, we developed eight binary classification models using different machine learning methods with 17 predictors. Development used data from 2013 to 2022, and performance was measured between 2017 and 2022 using a add-one-year-in expanding window approach. We used two calibration strategies: 95% sensitivity (High sensitivity) and optimizing the area under the curve (Balanced). A bootstrap estimated confidence intervals.
Setting The setting is a level one equivalent trauma center with bimonthly mortality and morbidity conferences for identifying OFIs; a combination of human review of individual patient cases and audit filters is used to screen patients for these conferences.
Participants A total of 8220 adult trauma patients were screened for OFI. All patients prompted trauma team activation or were later found to have an injury severity score greater than 9.
Main outcome measures Outcome measures were the models and audit filter performances, measured as discrimination, calibration, true positive and false positive rates.
Results OFI were identified in 496 (6%) patients. The best performing model was XGBoost (High sensitivity: [auc:0.75, sens:0.904, FPR: 0.599], and Balanced: [auc:0.75, sens:0.502, FPR: 0.186]) followed by Random Forest (High sensitivity: auc:0.733, sens:0.888, FPR: 0.617), and Balanced: [auc:0.733, sens:0.519, FPR: 0.222]). All machine learning models showed higher AUC and lower FPRs compared to Audit filters (auc:0.616, sens:0.903, FPR: 0.671).
Conclusion and Relevance Machine learning models generally outperformed audit filters in predicting OFI among adult trauma patients, balancing and reducing overall screening burden for trauma quality improvement programs while potentially identifying new OFI types.
Key point Question: How does the performance of machine learning models compare to that of audit filters when screening for opportunities for improvement (OFI), errors in care with adverse outcomes, among adult trauma patients?
Findings: Our registry-based cohort study including 8,220 patients showed that machine learning models outperform audit filters, exhibiting greater area under the curve values and reduced false-positive rates. Compared to audit filters, these models can be calibrated to balance sensitivity against overall screening burden.
Meaning: Machine learning models have the potential to reduce false positives when screening for OFI in adult trauma patients and thereby enhancing trauma quality programs.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
Supported by the Swedish Society of Medicine, grant number SLS-973387, and by The Swedish Carnegie Hero Fund.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The study was approved by Stockholm Research Ethics Review Board, approval number 2021-02541 and 2021-03531.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
Rewritten and changed methodology, results and discussion. Added active predictor choice. Measured performance over several years. Added calibration and two different approaches.
Data Availability
The data that support the findings of this study are available following the approval of a project suggesting to use the data by the Swedish Ethical Review Authority and the appropriate bodies at the Karolinska University Hospital. More information is available on request from the corresponding author, J. Attergrim.