RT Journal Article
SR Electronic
T1 Large-scale validation of the Prediction model Risk Of Bias ASsessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2021.01.20.21250183
DO 10.1101/2021.01.20.21250183
A1 Esmee Venema
A1 Benjamin S Wessler
A1 Jessica K Paulus
A1 Rehab Salah
A1 Gowri Raman
A1 Lester Y Leung
A1 Benjamin C Koethe
A1 Jason Nelson
A1 Jinny G Park
A1 David van Klaveren
A1 Ewout W Steyerberg
A1 David M Kent
YR 2021
UL http://medrxiv.org/content/early/2021/01/25/2021.01.20.21250183.abstract
AB Objective To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.Study Design and Setting We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation and assessed the change in discrimination (dAUC) between the derivation and the validation cohorts (n=1,147).Results PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 529/556 CPMs (95%) were classified as high ROB, 20 (4%) low ROB, and 7 (1%) unclear ROB. Median change in discrimination was significantly smaller in low ROB models (dAUC −0.9%, IQR −6.2%–4.2%) compared to high ROB models (dAUC −11.7%, IQR −33.3%–2.6%; p<0.001).Conclusion High ROB is pervasive among published CPMs. It is associated with poor performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.What is newHigh risk of bias is pervasive among published clinical prediction modelsHigh risk of bias identified with PROBAST is associated with poorer model performance at validationA subset of questions can distinguish between models with high and low risk of biasCompeting Interest StatementThe authors have declared no competing interest.Clinical TrialN/A not a clinical trialFunding StatementThis research was funded by a Patient-Centered Outcomes Research Institute (PCORI) Methods Award (ME-1606-35555).Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Tufts Health Sciences IRB #12461All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesWe used publications from the Tufts Predictive Analytics and Comparative Effectiveness (PACE) CPM Registry http://www.pacecpmregistry.org/