ABSTRACT
OBJECTIVE To compare performance and probability estimates of six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.
MATERIALS AND METHODS Models were developed on 5909 patients (recruited 1999-2012) and validated on 3199 patients (2012-2015). Nine clinical and ultrasound predictors were used. Outcome was based on histology following surgery within 120 days after the ultrasound examination. We developed models using multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM).
RESULTS Benign tumors were most common (62%), secondary metastatic tumors least common (5%). XGBoost, RF, NN and MLR had similar performance: c-statistics for benign versus any type of malignant tumors were 0.92, multiclass c-statistics 0.54-0.55, average Estimated Calibration Indexes 0.03-0.07, and Net Benefits at the 10% malignancy risk threshold 0.33-0.34. Despite poorer discrimination and calibration performance for Ridge MLR and in particular SVM, Net Benefits were similar for all models. The estimated probabilities often differed strongly between models. For example, the probability of a benign tumor differed by more than 20 percentage points in 29% of the patients, and by more than 30 percentage points in 16% of the patients.
DISCUSSION Several regression and machine learning models had very good and similar performance in terms of discrimination, calibration and clinical utility. Nevertheless, individual probabilities often varied substantially.
CONCLUSION Machine learning did not outperform MLR. The choice of algorithm can strongly affect probabilities given to a patient.
Competing Interest Statement
LV reported receiving grants from the Swedish Research Council, Malmö University Hospital and Skåne University Hospital, Allmänna Sjukhusets i Malmö Stiftelse för bekämpande av cancer (the Malmö General Hospital Foundation for Fighting Against Cancer), Avtal om läkarutbildning och forskning (ALF)–medel, and Landstingsfinansierad Regional Forskning during the conduct of the study; and teaching fees from Samsung outside the submitted work. DT and BVC reported receiving grants from the Research Foundation–Flanders (FWO) and Internal Funds KU Leuven during the conduct of the study. TB reported receiving grants from NIHR Biomedical Research Centre, speaking honoraria and departmental funding from Samsung Healthcare and grants from Roche Diagnostics, Illumina, and Abbott. No other disclosures were reported. All other authors declare no competing interests.
Funding Statement
The study is supported by the Research Foundation Flanders (FWO) (projects G049312N, G0B4716N, 12F3114N, G097322N), and Internal Funds KU Leuven (projects C24/15/037, C24M/20/064). DT is a senior clinical investigator of FWO, and WF was a clinical fellow of FWO. TB is supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare National Health Service (NHS) Trust and Imperial College London. The views expressed are those of the authors and not necessarily those of the NHS, NIHR, or Department of Health. LV is supported by the Swedish Research Council (grant K2014-99X-22475-01-3, Dnr 2013-02282), funds administered by Malmo University Hospital and Skane University Hospital, Allmanna Sjukhusets i Malmo Stiftelse for bekampande av cancer (the Malmo General Hospital Foundation for fighting against cancer), and two Swedish governmental grants (Avtal om lakarutbildning och forskning (ALF)-medel and Landstingsfinansierad Regional Forskning). The funders of the study had no role in the study design, data collection, data analysis, data interpretation, writing of the report, or in the decision to submit the paper for publication.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Ethics Committee of the University Hospitals Leuven gave ethical approval for this work (S64709)
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
