Abstract
Introduction Evaluation of artificial intelligence (AI) algorithms for prostate segmentation is challenging because ground truth is lacking. We aimed to (1) create a reference standard dataset with precise prostate contours by expert consensus and (2) evaluate various AI tools against this standard.
Materials and methods We obtained prostate MRI cases from six institutions from the Quantitative Prostate Imaging Consortium. A panel of four experts (two genitourinary radiologists, two prostate radiation oncologists) meticulously developed consensus prostate segmentations on axial T2-weighted series. We evaluated the performance of six AI tools (three commercially available, three academic) using Dice scores, distance from reference contour, and volume error.
Results The panel achieved consensus prostate segmentation on each slice of all 68 patient cases included in the reference dataset. We present two patient examples to serve as contouring guides. Depending on the AI tool, median Dice scores (across patients) ranged from 0.80 to 0.94 for whole prostate segmentation.
For a typical (median) patient, AI tools had a mean error over the prostate surface ranging from 1.3 to 2.4 mm. They maximally deviated 3.0 to 9.4 mm outside the prostate and 3.0 to 8.5 mm inside the prostate for a typical patient. Error in prostate volume measurement for a typical patient ranged from 4.3% to 31.4%.
Discussion We established an expert consensus benchmark for prostate segmentation. The best-performing AI tools have typical accuracy greater than that reported for radiation oncologists using CT scans (most common clinical approach for radiotherapy planning). Physician review remains essential to detect occasional major errors.
Competing Interest Statement
DJAM reports a clinical advisor role for Stratagen Bio and an ad hoc consultant role for Guerbet and Promaxo. SAW was supported by funding from the ARRS Scholarship for professional development and has received investigator-initiated research grants (paid to the institution) from Siemens. MC is employed by Radformation. JS is employed by Quibim. IV is employed by Cortechs.ai. The spouse of SCK is employed by Sanof. AMD is a founder of and holds equity interest in CorTechs Labs and serves on its scientific advisory board. He is also a member of the Scientific Advisory Board of Healthlytix and receives research funding from General Electric Healthcare (GEHC). TMS reports honoraria from Multimodal Imaging Services Corporation, Varian Medical Systems, Janssen, and WebMD; he has an equity interest in CorTechs Labs, Inc. and serves on its Scientific Advisory Board. These companies might potentially benefit from the research results. The terms of the above arrangements have been reviewed and approved by the University of California San Diego in accordance with its conflict- of -interest policies The remaining authors have no conflicts of interest to disclose.
Funding Statement
National Institutes of Health, Grant/AwardNumbers: NIH/NIBIB K08EB026503, NIHUL1TR000100; American Society for Radiation Oncology; Prostate Cancer Foundation; Department of Defense,Grant/Award Number: DOD/CDMRPPC220278
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The IRBs of University of California San Diego, Massachusetts General Hospital, University of Rochester Medical Center, University of California San Francisco, University of Texas Health Sciences Center San Antonio, and University of Cambridge gave ethical approval for this work.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
