RT Journal Article
SR Electronic
T1 A Systematic Review of Machine Learning-based Prognostic Models for Acute Pancreatitis: Towards Improving Methods and Reporting Quality
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2024.06.26.24309389
DO 10.1101/2024.06.26.24309389
A1 Critelli, Brian
A1 Hassan, Amier
A1 Lahooti, Ila
A1 Noh, Lydia
A1 Park, Jun Sung
A1 Tong, Kathleen
A1 Lahooti, Ali
A1 Matzko, Nate
A1 Adams, Jan Niklas
A1 Liss, Lukas
A1 Quion, Justin
A1 Restrepo, David
A1 Nikahd, Melica
A1 Culp, Stacey
A1 Lacy-Hulbert, Adam
A1 Speake, Cate
A1 Buxbaum, James
A1 Bischof, Jason
A1 Yazici, Cemal
A1 Phillips, Anna Evans
A1 Terp, Sophie
A1 Weissman, Alexandra
A1 Conwell, Darwin
A1 Hart, Phil
A1 Ramsey, Mitch
A1 Krishna, Somashekar
A1 Han, Samuel
A1 Park, Erica
A1 Shah, Raj
A1 Akshintala, Venkata
A1 Windsor, John A
A1 Mull, Nikhil K
A1 Papachristou, Georgios I
A1 Celi, Leo Anthony
A1 Lee, Peter J
YR 2024
UL http://medrxiv.org/content/early/2024/06/27/2024.06.26.24309389.abstract
AB Background An accurate prognostic tool is essential to aid clinical decision making (e.g., patient triage) and to advance personalized medicine. However, such prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However, methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical to model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models.Methods Using a validated search strategy, we identified ML AP studies from the databases MEDLINE, PubMed, and EMBASE published between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool (PROBAST). Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis – Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models.Results The search strategy identified 6480 publications of which 30 met the eligibility criteria. Studies originated from China (22), U.S (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area-under-the-curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed.Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human-AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30) and reporting source data (19/30),.Discussion There are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility and implementation of these prognostic models despite their promise of superior predictive accuracy.Funding noneRegistration Research Registry (reviewregistry1727)Competing Interest StatementNO authors have competing interestsClinical Protocols https://doi.org/10.1186/s41512-024-00169-1 https://pubmed.ncbi.nlm.nih.gov/38561864/ Funding StatementThe author(s) received no specific funding for this work.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.Yeshttps://figshare.com/s/64a07bd4eb2a0f334e69 DOI: 10.6084/m9.figshare.26078743 https://figshare.com/s/64a07bd4eb2a0f334e69 https://doi.org/10.6084/m9.figshare.26078743