PT  - JOURNAL ARTICLE
AU  - Yuan, Kevin
AU  - Luk, Augustine
AU  - Wei, Jia
AU  - Walker, A Sarah
AU  - Zhu, Tingting
AU  - Eyre, David W
TI  - Machine learning to predict antibiotic susceptibility in Enterobacterales bloodstream infections compared to clinician prescribing
AID  - 10.1101/2024.10.02.24314776
DP  - 2024 Jan 01
TA  - medRxiv
PG  - 2024.10.02.24314776
4099  - http://medrxiv.org/content/early/2024/10/02/2024.10.02.24314776.short
4100  - http://medrxiv.org/content/early/2024/10/02/2024.10.02.24314776.full
AB  - Background Patients with Gram-negative bloodstream infections are at risk of serious adverse outcomes without active treatment, but identifying who has antimicrobial resistance (AMR) to target empirical treatment is challenging.Methods We used XGBoost machine learning models to predict the presence of antimicrobial resistance to seven antibiotics in patients with Enterobacterales bloodstream infection. Models were trained using hospital and community data available at the time blood cultures were obtained from Oxfordshire, UK, between 01-January-2017 and 31-December-2021. Model performance was compared to final microbiology results using test datasets from 01-January-2022 to 31-December-2023 and with clinicians’ prescribing.Findings 4709 infection episodes were used for model training and evaluation; antibiotic resistance rates ranged from 7-67%. In held-out test data, resistance prediction performance was similar for the seven antibiotics (AUCs 0.680 [95%CI 0.641-0.720] to 0.737 [0.674-0.797]). Performance improved for most antibiotics when species data were included as model inputs (AUCs 0.723 [0.652-0.791] to 0.827 [0.797-0.857]). In patients treated with a beta-lactam, clinician prescribing led to 70% receiving an active beta-lactam: 44% were over-treated (broader spectrum treatment than needed), 26% optimally treated (narrowest spectrum active agent), and 30% under-treated (inactive beta-lactam). Model predictions without species data could have led to 79% of patients receiving an active beta-lactam: 45% over-treated, 34% optimally treated, and 21% under-treated.Interpretation Predicting AMR in bloodstream infections is challenging for both clinicians and models. Despite modest performance, machine learning models could still increase the proportion of patients receiving active empirical treatment by up to 9% over current clinical practice in an environment prioritising antimicrobial stewardship.Funding National Institute of Health Research (NIHR) Oxford Biomedical Research Centre, NIHR Health Protection Research Unit in Healthcare-associated Infection and Antimicrobial Resistance.Evidence before this study We searched Pubmed and Google Scholar using the terms: [antibiotic OR antimicrobial] AND [resistance] AND [prediction OR machine learning OR AI OR artificial intelligence] for articles published up to 31 August 2024. References and citations for articles identified were also reviewed. Several studies have shown that machine learning can potentially be used to predict antimicrobial resistance (AMR) subsequently identified on phenotypic antimicrobial susceptibility testing. Most have focused either on identifying resistance in urinary tract infection, or in all samples received by a microbiology laboratory, which are often dominated by urine cultures. Only two studies were identified focusing specifically on bloodstream infection, and these only investigated a limited number of antibiotics. Overall, prediction performance was typically modest, e.g. area under the receiver operating curve (AUC) values of 0.65-0.75. Most studies focus on data available in the community or hospital but not both. Four studies retrospectively compared clinical prescribing to model predictions and showed models could potentially reduce inappropriate antibiotic use, but none focused specifically on bloodstream infection. External validation of models is uncommon, and most studies do not cover how models can be updated over time or to new locations.Added value of this study We developed machine learning models to predict resistance to seven antibiotics (amoxicillin, co-amoxiclav, ceftriaxone, piperacillin-tazobactam, ciprofloxacin, co-trimoxazole, and gentamicin) in bloodstream infections caused by Enterobacterales species. We focused on this clinical syndrome as it is an important cause of AMR-associated mortality. We used data from Oxfordshire, UK, between January 2017 and December 2023 for model training and evaluation (4709 infection episodes in 4243 patients). In held-out test data, predictive performance was similar for the seven antibiotics (AUCs 0.680 [95%CI 0.641-0.720] to 0.737 [0.674-0.797]). Performance improved for most antibiotics when species data were included as model inputs (AUCs 0.723 [0.652-0.791] to 0.827 [0.797-0.857]). AMR identified in recent microbiology results was the most important predictor of resistance. Model performance was relatively consistent over time. AMR prediction was also challenging for clinicians: their implied sensitivity for detecting resistance, i.e., the proportion of patients treated with a beta-lactam with resistance receiving active treatment was 97% for amoxicillin, 29% for co-amoxiclav, 19% for ceftriaxone, and 6% for piperacillin-tazobactam. In patients treated with a beta-lactam, clinician prescribing led to 70% receiving an active beta-lactam: 44% were over-treated (broader spectrum treatment than needed), 26% optimally treated (narrowest spectrum active agent), and 30% under-treated (inactive beta-lactam). Model predictions without species information could have led to 79% of patients receiving an active beta-lactam: 45% over-treated, 34% optimally treated, and 21% under-treated.Implications of all the available evidence Despite considering a wide range of input features, including hospital and some community data, model performance was broadly consistent with what has been described previously for similar tasks. This suggests there is a potential ceiling on the performance of machine learning in this context. However, despite modest performance, machine learning models could still increase the proportion of patients receiving active treatment by up to 9% over current clinical practice in an environment prioritising antimicrobial stewardship.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study was funded by the National Institute for Health Research (NIHR) Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Oxford University in partnership with the UK Health Security Agency (UKHSA) and the NIHR Biomedical Research Centre, Oxford. DWE is supported by a Robertson Fellowship. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research, the Department of Health or the UKHSA.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Deidentified data were obtained from Infections in Oxfordshire Research Database (IORD), which has approvals from the South Central-Oxford C Research Ethics Committee (19/SC/0403), the Health Research Authority and the Confidentiality Advisory Group (19/CAG/0144) as a deidentified database without individual consent.I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesThe datasets analysed during the current study are not publicly available as they contain personal data but are available from the Infections in Oxfordshire Research Database (https://oxfordbrc.nihr.ac.uk/ research-themes-overview/antimicrobial-resistance-and-modernising-microbiology/infections-in-oxfordshire-research-database-iord/), subject to an application and research proposal meeting the ethical and governance requirements of the Database. For further details on how to apply for access to the data and for a research proposal template please email iord{at}ndm.ox.ac.uk.