PT  - JOURNAL ARTICLE
AU  - Michael Ko
AU  - Emma Chen
AU  - Pranav Rajpurkar
AU  - Ashwin Agrawal
AU  - Anand Avati
AU  - Andrew Ng
AU  - Sanjay Basu
AU  - Nigam H. Shah
TI  - Improving Hospital Readmission Prediction using Individualized Utility Analysis
AID  - 10.1101/2020.07.26.20156943
DP  - 2020 Jan 01
TA  - medRxiv
PG  - 2020.07.26.20156943
4099  - http://medrxiv.org/content/early/2020/07/29/2020.07.26.20156943.short
4100  - http://medrxiv.org/content/early/2020/07/29/2020.07.26.20156943.full
AB  - Importance Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources.Objective To determine whether ML models for allocating readmission-mitigating interventions are ranked differently based on their overall utility and their discriminative ability.Design A retrospective analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart.Setting Health plan claims from all 50 states for commercially-insured individuals.Participants 513,495 patients who were admitted as inpatients over the period January 2016 through January 2017.Main Outcomes and Measures Maximum utility achieved by three machine learning models for allocating readmission-mitigating interventions, determined using cost accrued in the 90 days post-discharge of an index admission and estimated counterfactual cost. Data were analyzed between April 2019 and March 2020.Results The study sample consisted of 513,495 patients (mean [SD] age 69 [19] years; 294,895 [57%] Female) mean 90 day cost of $11,552 for the study period. Allocating readmission-mitigating interventions based on a LightGBM model trained to predict readmissions achieved a maximum utility of $-12,645 per patient, and an AUC of 0.74 (95% CI 0.74, 0.75); allocating interventions based on a model trained to predict cost as a proxy achieved a higher maximum utility of $-12,472 per patient, and an AUC of 0.63 (95% CI 0.62, 0.63). A hybrid model combining both intervention strategies achieved a maximum utility of $-12,472, and an AUC of 0.71 (95% CI 0.71, 0.71), comparable with the best models on either metric.Conclusion and Relevance We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.Question Do machine learning models for allocating readmission-mitigating interventions rank differently based on overall utility and discriminative performance?Finding A machine learning model predicting a patient’s future cost of care was able to achieve higher utility than a readmission risk prediction model, even though it had a lower discriminative performance in predicting readmissions.Meaning Our study suggests that machine learning models guiding allocation of limited health resources may consider evaluating and optimizing on utility.Competing Interest StatementThe authors have declared no competing interest.Funding StatementNo funding recievedAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:The Stanford University administrative panel for the Protection of Human Subjects approved this study.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesNA