Study Protocol: Development and Retrospective Validation of an Artificial Intelligence System for Diagnostic Assessment of Prostate Biopsies

7.1.1. Karolinska University Hospital (KUH-1)

The KUH-1 samples were collected at the Department of Pathology, Karolinska University Hospital in Solna, Sweden in 2018. Among the cases assessed by L.E. during 2018, we included all positive slides of all patients diagnosed with ISUP grade 2-5 cancer, all positive slides from a random selection of patients diagnosed with ISUP grade 1 cancer, and all slides from a random selection of patients with a negative diagnosis. Patients underwent systematic transrectal biopsies in approximately 1/3 of the cases, and magnetic resonance imaging (MRI) targeted or combined biopsies in approximately 2/3 of cases. Slides typically contain one core, sectioned at two levels. This cohort has been used as an external validation set in previous studies (Ström et al., 2020; Bulten et al., 2022).

7.1.2. Radboud University Medical Center (RUMC)

The RUMC samples were collected at the Radboud University Medical Center in Nijmegen, the Netherlands from January 2012 to December 2017 (Bulten et al., 2020). Patients were sampled randomly, stratified by the highest reported GS in the pathology reports, and the slide with the most aggressive part of the tumour was included for each patient. Additionally, a group of patients with only benign biopsies were randomly sampled. Patients generally underwent MRI-targeted transrectal biopsy. The data underwent additional refinement in preparation for the PANDA Kaggle challenge (Bulten et al., 2022): only one core, sectioned at one level was retained per WSI, the background was masked to hide most of the markings made on the glass, and the images were converted into .tiff format (JPEG compression, quality 70). For the purposes of PANDA, the cohort was partitioned into three sets—development, tuning, and internal validation, stratified by patient and the highest Gleason pattern in the biopsy.

7.1.3. Capio S:t Göran Hospital (STG)

The STG samples were collected at Capio S:t Göran Hospital in Stockholm, Sweden from 2016 to 2017. We included a random selection of slides with an enrichment for high-grade cancer. Patients underwent transrectal biopsy, and slides typically contain one core, sectioned at two levels. This cohort was also part of the development set in a previous study (Ström et al., 2020).

7.1.4. Stockholm3 (STHLM3)

The STHLM3 samples were collected in a population-based clinical trial (ISRCTN84445406) (Grönberg et al., 2015) from 2012 to 2015 in Stockholm, Sweden. Histological sample preparation was performed at Histocenter, Gothenburg, Sweden, and the samples were assessed at the Department of Pathology, Karolinska University Hospital in Stockholm. Patients underwent 10-12 core systematic transrectal biopsies and slides usually contain one core, sectioned at two levels. Subsets of the digitised samples have been used as development and internal validation sets in previous studies (Ström et al., 2020; Bulten et al., 2022; Ji et al., 2022; Kartasalo et al., 2022; Olsson et al., 2022). Patient and slide selection, retrieval and digitisation took place on five occasions between 2014 and 2023 (see Table 2), as below:

• 2014: All cores from the first 500 patients diagnosed with prostate cancer in the STHLM3 trial were scanned on a Hamamatsu NanoZoomer 2.0-HT.

• 2017-2019: All patients with at least one core graded as GS 4 + 4 or 5 + 5 and 497 randomly selected patients with at least one core graded as 3 + 3 were considered. From each of these patients, we included all positive cores and a randomly selected negative core. Finally, we randomly selected 139 cancer-free patients from whom we included one randomly selected core. Additionally, we added all cores which were indicated to have PNI and had not been scanned earlier. The cores were scanned on an Aperio AT2.

• 2018-2019: The cores of a random selection of patients were scanned on a Hamamatsu NanoZoomer XR.

• 2019-2020: The cores of a random selection of patients were scanned on the Philips IntelliSite Ultra Fast Scanner (UFS).

• 2023: Patients belonging to the PANDA challenge Swedish public and private validation sets were scanned on the Grundium Ocus40.

• 2023: Initially, cores with < 4 millimetres of cancer were excluded to have sufficient cancer tissue for future molecular profiling of the samples. Among the remaining patients, 50% of those with ISUP 1 or ISUP 2 (patient level ISUP) were randomly selected for inclusion, while all patients with ISUP 3-5 were included for scanning on the Grundium Ocus40.

7.1.5. Stavanger University Hospital (SUH)

The SUH samples represent consecutive cases collected from routine diagnostics at the Department of Pathology, Stavanger University Hospital in Stavanger, Norway from December 2016 to March 2018. Biopsies were taken at the Department of Urology in Stavanger University Hospital and other private urological clinics at the Stavanger Urological Center. Patients primarily underwent systematic transrectal biopsies, although some received MRI-targeted biopsies, either alone or combined with systematic biopsy. Slides typically contain two cores from the same anatomical location, sectioned at two levels. A subset of the SUH cohort has been used as an external validation set in previous studies (Ji et al., 2022; Olsson et al., 2022).

7.2.1. Aichi Medical University (AMU)

The AMU samples were collected at the Aichi Medical University in Nagakute, Japan from 2020 to 2023. Samples were selected to include cribriform prostate cancer cases and non-cribriform cases. Cribriform cases were chosen sequentially, while non-cribriform cases were selected among cases containing Gleason pattern 4 and age-adjusted to match the cribriform cases. Patients generally underwent systematic transrectal biopsy, with only a few undergoing MRI-targeted biopsy. Slides typically contain several cores, sectioned at several levels.

7.2.1.1. Reference standard protocol

All cases were assessed by a uropathologist (T.T.) initially using a microscope and then confirmed digitally with the NDP.View software using Hamamatsu WSIs (.ndpi). The presence or absence of cribriform prostate cancer was reported on slide level and GS was reported on patient level.

7.2.2.1. Reference standard protocol

A uropathologist (H.S.) assessed the slides microscopically and reported the GS, ISUP grade, additional stainings (e.g. IHC), and the presence or absence of specific morphological subtype categories on slide level (see Table 5). Slides representing benign mimickers were microscopically re-assessed by the lead pathologist (L.E.).

7.2.3.1. Reference standard protocol

All cases were assessed by a uropathologist (B.P.U.) microscopically and the GS, the ISUP grade, cancer extent and biopsy length were reported on the slide level.

Subsets of the AUH cohort underwent additional re-assessments as follows:

• A subset of slides (n=41) was randomly selected, stratified by the ISUP grade, for re-assessment by the lead pathologist (L.E.). This re-assessment was conducted digitally on Cytomine using Hamamatsu WSIs (.ndpi) to report the GS per slide.

7.2.4.1. Reference standard protocol

The reference standard protocol for the KUH-2 cohort adheres to KUH-1, except for additional reporting of the presence or absence of specific morphological subtype categories, assessed by the lead pathologist (L.E.) on slide level (see Table 5).

7.2.5.1. Reference standard protocol

The reference standard was obtained from routine assessments done by several pathologists using a microscope to determine the GS, the ISUP grade, cancer extent and biopsy length per patient or per anatomical location (i.e. a set of biopsy cores assessed together).

Subsets of the MLP cohort underwent additional re-assessments as follows:

• A subset of slides (n=66) was randomly selected, stratified by the ISUP grade, for re-assessment by the lead pathologist (L.E.). The patient level ISUP grade was used for stratification, due to missing slide level grading. This re-assessment was conducted digitally on Cytomine using 3DHISTECH WSIs (.mrxs) to report the GS per slide.

7.2.6.1. Reference standard protocol

The reference standard was determined based on an initial assessment by a single pathologist (M.B.) and a second review by a more experienced pathologist (R.K.). Both pathologists have a specialisation in surgical pathology and are currently specialising in uropathology. The pathologists assessed the cases using a microscope and reported the GS, the ISUP grade, total cancer percentage and Gleason pattern 4 and 5 percentages on the slide level.

Subsets of the MUL cohort underwent additional re-assessments as follows:

• A subset of slides (n=66) was randomly selected, stratified by ISUP grade, for re-assessment by the lead pathologist (L.E.). This re-assessment was conducted digitally on Cytomine using Grundium WSIs (.svs) to report the GS per slide.

• All slides containing Gleason pattern 4 (n=276) were initially assessed for potential cribriform cancer by a uropathologist (A.B.). The assessment was conducted digitally on Cytomine using Grundium WSIs (.svs) to report cribriform cancer per slide and mark the positive and borderline foci. All foci were then re-assessed on Cytomine by the lead pathologist (L.E.).

• The slides (n=276) assessed for cribriform cancer were also initially assessed for potential PNI by a uropathologist (A.B.). The assessment was conducted on Cytomine using Grundium WSIs (.svs) to report PNI per slide and mark the positive and borderline foci. All foci were then re-assessed on Cytomine by the lead pathologist (L.E.).

7.2.7.1. Reference standard protocol

The reference standard was determined based on the pathology reports from routine diagnostics. Using the microscope the pathologists reported the GS, the ISUP grade, cancer extent, biopsy length, Gleason pattern 4 percentage, cribriform cancer, PNI, high-grade prostatic intraepithelial neoplasia (HGPIN) and possible IHC staining per anatomical location (i.e. a set of biopsy cores assessed together) and per patient.

Subsets of the SCH cohort underwent additional re-assessments as follows:

• A subset of slides (n=72) was randomly selected, stratified by the ISUP grade and anatomical location for re-assessment by the lead pathologist (L.E.). This re-assessment was conducted digitally on Cytomine using Philips WSIs (.isyntax converted to .tiff) to report the GS per slide.

• A subset of slides (n=56) were digitally re-assessed for cribriform cancer by a uropathologist (H.S.). We selected all positive anatomical locations and a random selection of 6 negative anatomical locations with Gleason pattern 4 tissue and included all slides from these locations. This re-assessment was conducted digitally on Cytomine using Philips WSIs (.isyntax converted to .tiff) to report cribriform cancer per slide. The pathologist could also indicate uncertain cases with a borderline category.

• A subset of slides (n=94) were digitally re-assessed for PNI by a uropathologist (B.D.). We randomly selected 12 positive and 5 negative anatomical locations per ISUP grade and included all slides from these locations. This re-assessment was conducted digitally on Cytomine using Philips WSIs (.isyntax converted to .tiff) to report PNI per slide. The pathologist could also indicate uncertain cases with a borderline category.

7.2.8.1. Reference standard protocol

The reference standard was determined based on the pathology reports from routine diagnostics. Using the microscope the pathologists reported the GS, the ISUP grade, cancer extent, biopsy length, Gleason pattern 4 percentage, cribriform cancer, PNI, HGPIN and possible IHC staining per anatomical location (i.e. a set of biopsy cores assessed together) and in some cases per patient.

Subsets of the SFI cohort underwent additional re-assessments as follows:

• A subset of slides (n=67) was randomly selected, stratified by the ISUP grade and anatomical location for re-assessment by the lead pathologist (L.E.). This re-assessment was conducted digitally on Cytomine using Philips WSIs (.isyntax converted to .tiff) to report the GS per slide.

7.2.9.1. Reference standard protocol

The reference standard was determined based on the pathology reports from routine diagnostics. Pathologists using a microscope reported the GS, the ISUP grade, cancer extent, biopsy length, Gleason pattern 4 percentage, cribriform cancer, PNI, HGPIN and possible IHC staining per anatomical location (i.e. slide) and in some cases per patient.

Subsets of the SFR cohort underwent additional re-assessments as follows:

• A subset of slides (n=49) was randomly selected, stratified by the ISUP grade and anatomical location for re-assessment by the lead pathologist (L.E.). This re-assessment was conducted digitally on Cytomine using Philips WSIs (.isyntax converted to .tiff) to report the GS per slide.

7.2.10.1. Reference standard protocol

The reference standard was obtained from the clinical routine. The pathologists assessed the slides microscopically and reported the ISUP grade at the patient level in two ways: as the maximum and as the average of the slide level ISUP grades. The underlying slide-level ISUP grades were not provided on the AIDA Data Hub.

Subsets of the SPROB20 cohort underwent additional re-assessments as follows:

• A subset of slides (n=50) was randomly selected, stratified by ISUP grade and patient, for re-assessment by the lead pathologist (L.E.). This re-assessment was conducted digitally on Cytomine using Hamamatsu WSIs (.ndpi converted to .tiff) to report the GS per slide.

7.2.11.1. Reference standard protocol

The reference standard was determined digitally by a panel of 10 different pathologists from Austria, Germany, Israel, Japan, the Netherlands, Russia and the United States. All pathologists reported the ISUP grade per slide and the final grade was obtained as the majority vote. A consensus was considered reached in cases where the majority ISUP grade had at least six votes.

7.2.12.1. Reference standard protocol

The reference standard was determined digitally by a panel of 11 different pathologists from Austria, Germany, Israel, Japan, the Netherlands, Russia and the United States. All pathologists reported the ISUP grade per slide and the final grade was obtained as the majority vote. A consensus was considered reached in cases where the majority ISUP grade had at least six votes.

8.1.1. Primary analysis: Diagnosis and Gleason scoring

1. Internal and external validation against the original cohort-specific reference standard

2. Subgroup analyses

   1. Evaluate performance across different age groups

   2. Evaluate performance on systematic vs. targeted biopsies

   3. Evaluate performance on non-treated patients vs. patients treated for benign prostatic hyperplasia prior to biopsy

   4. Evaluate performance on morphological subtypes

   5. Evaluate performance on cases requiring vs. not requiring IHC staining

   6. Evaluate performance compared to the current state-of-the-art AI systems

3. Sensitivity analyses

   1. Cross-scanner consistency analyses

   2. Compare the AI system vs. individual pathologist panel members

   3. Internal and external validation against uniform reference standard by the lead pathologist

   4. Blinded re-assessment of slides with marked errors

8.1.2. Secondary analysis: Cancer extent prediction

1. Internal and external validation against the original cohort-specific reference standards

2. Subgroup analyses

   1. Evaluate performance across different age groups

   2. Evaluate performance on systematic vs. targeted biopsies

   3. Evaluate performance on non-treated patients vs. patients treated for benign prostatic hyperplasia prior to biopsy

3. Sensitivity analyses

   1. Cross-scanner consistency analyses

8.1.3. Secondary analysis: Cribriform cancer detection

1. Internal and external validation against the original cohort-specific reference standards

2. Subgroup analyses

   1. Evaluate performance across different age groups

   2. Evaluate performance on systematic vs. targeted biopsies

   3. Evaluate performance on non-treated patients vs. patients treated for benign prostatic hyperplasia prior to biopsy

3. Sensitivity analyses

   1. Cross-scanner consistency analyses

   2. Compare the AI system vs. individual pathologist panel members

   3. Re-assessment excluding borderline slides

8.1.4. Secondary analysis: Perineural invasion detection

1. Internal and external validation against the original cohort-specific reference standards

2. Subgroup analyses

   1. Evaluate performance across different age groups

   2. Evaluate performance on systematic vs. targeted biopsies

   3. Evaluate performance on non-treated patients vs. patients treated for benign prostatic hyperplasia prior to biopsy

3. Sensitivity analyses

   1. Cross-scanner consistency analyses

   2. Compare the AI system vs. individual pathologist panel members

   3. Re-assessment excluding borderline slides

8.1.5. Exploratory analyses

1. Evaluate visualisations of the AI output

2. Evaluate the impact of tissue segmentation algorithms

3. Evaluate end-to-end vs. transfer-learning-based models

4. Evaluate the impact of physical colour calibration

Primary analysis: Diagnosis and Gleason scoring

We will quantify the concordance of the AI system’s cancer diagnosis (positive/negative), Gleason score and ISUP grade with the reference standards in the tuning, internal validation and external validation cohorts using the metrics described below. The analysis will be conducted on slide level (AQ, AUH, KUH-1, KUH-2, MUL, RUMC, SFR, STHLM3, SUH, UKK, WNS), anatomical location level (MLP, SFI, SCH) and/or patient level (KUH-1, SCH, SFI, SFR, SPROB20) depending on the granularity of the available reference standards.

Secondary analysis: Cancer extent prediction

We will quantify the concordance of the AI system’s prediction of linear cancer extent expressed in millimetres with the reference standards in those tuning, internal validation and external validation cohorts where a reference standard is available (AUH, KUH-1, STHLM3, SUH, STG, MLP, SCH, SFI, SFR). The concordance will be quantified using root mean squared error (RMSE). In addition, we will also report Pearson’s linear correlation coefficient, and show scatter plots of predicted millimetre cancer length vs. millimetre cancer length reported by the reference standard. The analysis will be conducted on slide level (AUH, KUH-1, STHLM3, SUH, STG, SFR), anatomical location level (MLP, SFI, SCH) and/or patient level (MLP, SCH, SFI, SFR) depending on the granularity of the available reference standards (see Table 3). Confidence intervals will be computed using the non-parametric bootstrap over cases.

Secondary analysis: Cribriform cancer detection

We will quantify the concordance of the AI system’s prediction of the presence of cribriform cancer with the reference standards in those internal and external validation cohorts where a reference standard is available (MUL, SCH, STHLM3, SUH). The tuning set has an insufficient number of cribriform samples for evaluation and will be included in the training. The concordance will be quantified using unweighted Cohen’s kappa. In addition, we will also report AUROC, sensitivity (true positive rate), specificity (true negative rate) and confusion matrices. Slides reported as borderline for cribriform cancer will be considered negative. The analysis will be conducted on slide level. Confidence intervals will be computed using the non-parametric bootstrap over cases.

Secondary analysis: Perineural invasion detection

We will quantify the concordance of the AI system’s prediction of the presence of perineural invasion with the reference standards in those internal and external validation cohorts where a reference standard is available (MUL, SCH, STHLM3, SUH). The tuning set has an insufficient number of PNI samples for evaluation and will be included in the training. The concordance will be quantified using unweighted Cohen’s kappa. In addition, we will also report AUROC, sensitivity (true positive rate), specificity (true negative rate) and confusion matrices. Slides reported as borderline for perineural invasion will be considered negative. The analysis will be conducted on slide level. Confidence intervals will be computed using the non-parametric bootstrap over cases.

Subgroup analyses

Sensitivity analyses

Exploratory analysis: Evaluate visualisations of the AI output

We will output visualisations of the AI system’s predictions to highlight areas on each slide containing different Gleason patterns, cribriform cancer or PNI. The visualisations will be assessed qualitatively by the lead pathologist (L.E.) and/or other experienced uropathologists for concordance with their assessments. We may additionally quantify the rate of agreement between the AI system and the pathologists by collecting region annotations to serve as a reference standard, and by calculating the pixel-wise sensitivity, specificity, intersection over union or other suitable metrics.

Exploratory analysis: Evaluate the impact of tissue segmentation algorithms

Detecting tissue from the background to only apply the rest of the analysis on tissue pixels is a common preprocessing step for most computational pathology algorithms. While this task of tissue segmentation may seem trivial, many modern AI algorithms reach such low error rates in their main task, that any errors in tissue detection can contribute to the overall model performance in a considerable way. In particular, missed tissue poses a risk of false negative diagnoses, if this leads to the exclusion of malignant tissue from the analysis. We will evaluate the effect of tissue segmentation on the overall performance of the AI system in terms of the primary and secondary objectives by comparing two different tissue segmentation algorithms. One of the algorithms represents classical image processing and relies on filtering and thresholding the image (Ström et al., 2020). The other algorithm is a trained deep learning based segmentation model. We will apply both algorithms to perform the tissue segmentation during model training and validation and compare the results on the internal and external validation cohorts.

Exploratory analysis: Evaluate end-to-end vs. transfer-learning-based models

Recently, so-called foundation models trained in a self-supervised manner on large and heterogeneous datasets, have been proposed as generally applicable solutions to diverse tasks in computational pathology as an alternative to tissue type or task specific models (Chen et al., 2024). We aim to compare our end-to-end trained prostate cancer specific model to transfer-learning-based models relying on state-of-the-art foundation models for histopathology. We will apply a suitable foundation model as a feature extractor and train an additional classifier to adapt the model to the task of diagnosis and Gleason scoring of prostate biopsies. For this transfer learning step, we will use the same development cohorts as for the end-to-end trained model. We will then evaluate the model on the same internal and external validation cohorts as the end-to-end trained model for a direct comparison.

Exploratory analysis: Evaluate the impact of physical colour calibration

Variations in the reproduction of colour across different digital pathology scanners may pose a problem for AI, leading to inconsistent model outputs depending on the scanner used for slide digitisation. A physical calibrant in the form of a spectrophotometrically characterised slide has been proposed as a means for standardising the colour characteristics of WSIs acquired with different scanners (Clarke et al., 2018). We will evaluate the impact of applying physical colour calibration on the performance of the AI model on those internal and external validation cohorts where the calibrant slide could be scanned on the same scanner as the prostate biopsies to allow calibration.

Retrieval and digitisation of clinical/pathology data

Depending on the data cohort, the clinical and pathology data were extracted from existing databases/registries (STHLM3) in tabular form, provided in tabular form by the data providing sites (AMU, AQ, AUH, MLP, MUL, RUMC, SPROB20, SUH, UKK, WNS) or tabulated manually in-house from pathology reports scanned into PDF files (KUH-1, KUH-2, SCH, SFI, SFR, STG). The manual tabulation in-house involved human translation of the reports from Finnish (SFI), French (SCH, SFR) and Swedish (KUH-1, KUH-2, STG) by trained non-experts fluent in the respective languages. Patient identifiers were pseudonymised during the data extraction or tabulation process by each data provider.

Retrieval and digitisation of slides

Slides were retrieved from the respective archives at each site and scanned with the instruments tabulated in Table 2. Each slide had a label with an identifier and depending on the scanning site, the identifiers were stored either in the form of macro/label images as part of the WSI metadata, automatically detected from QR codes and stored as WSI metadata, or manually typed in by the scanner operator when naming the resulting WSI files.

Linking slides to clinical/pathology data

Depending on the manner in which slide identifiers were stored for each WSI, the linking step involved one of the following approaches. For WSIs, where the identifier was manually typed into the filename, customised scripts were written in Python for each data cohort to parse the filename strings. This involved comparing the parsed identifiers to those present in the clinical/pathology data, and iterative refinements to rectify issues such as missing or additional zeros, missing or additional whitespace or other delimiters, and discrepancies with the representation of characters not belonging to the Basic Latin (standard ASCII) set, e.g. Ä or Ö. For WSIs, where the identifier was stored in the form of WSI metadata, we used an in-house developed optical character recognition (OCR) system to extract identifiers in a semi-automated manner from the QR-code based metadata items and the macro/label images embedded in the WSIs. The system first extracted the QR-code based identifier, if available, or performed OCR using the pytesseract (version 0.3.2) implementation of the Tesseract OCR engine (Smith, 2007). The system featured a simple user interface, which presented the automatically detected identifier pre-filled into a text box, alongside the macro/label image of the slide. The human operator then had the option of accepting the proposed identifier or correcting it manually based on the label image. All identifiers were assessed by trained non-experts using this semi-automated approach.

Relabeling

Slides and patients were initially labelled independently by each data provider using pseudonymised identifiers. This poses a risk that the same identifier (e.g. Patient_01) is used by multiple data providers, which would cause ambiguous matches in the final combined dataset. In order to minimise this risk and to obtain unique identifiers for each WSI, each slide and each patient, we calculated unique MD5 hashes based on the variables below. This step additionally provided another round of pseudonymisation to minimise the risk of any non-pseudonymised identifiers being accidentally used by the data providing sites.

• WSI ID: Filename + scanner serial number + scanning time stamp

• Slide ID: Cohort name + original slide ID

• Patient ID: Cohort name + original patient ID

Verification

The final dataset covering all the data cohorts is managed internally as a CSV spreadsheet, generated and maintained using scripts written in Python relying on pandas (Creators The pandas development team; McKinney, 2010). Upon generation and any modifications, the dataset undergoes comprehensive unit testing to ensure correctness, implemented in Python using the unittest framework. A version history of the dataset is retained to allow tracing back errors. In summary, the tests used for verification cover the following aspects. The uniqueness and unambiguity of matches based on the identifiers described above are verified. Patient-level variables are tested for consistency across all slides and WSIs from the

same patient, and slide level variables are checked for consistency across multiple WSIs representing the same slide. We verify that all variables have valid values, with specific tests for categorical, quantitative, and Boolean variables and test for logical mismatches between variables (e.g. a slide negative for cancer cannot be positive for PNI). We ensure there is no overlap between patients in different development vs. validation splits or between cross-validation folds in the development data. Please refer to the Supplementary Appendix Section 2 for an extensive list of all the tests.