Large-Scale Validation Study of an Improved Semi-Autonomous Urine Cytology Assessment Tool: AutoParis-X

Specimen Collection and Slide Processing

A total of 1,303 urine specimens were collected across 140 bladder cancer patients (median of 8 specimens per patient; IQR: [8-13]) from 2008 to 2019 at Dartmouth-Hitchcock Medical Center. Forty-seven of these specimens were used to curate data for training the cell and cluster-level machine learning models (cell and cluster-level training and validation cohort). Four specimens were removed due to equivocal findings and/or excessive confluent cellularity. AutoParis-X was further trained and validated on 1,252 specimens after curating slide-level cell/cluster predictors (slide-level training and validation cohorts; see Calculation of Cell and Cluster Slide-Level Scores). The specimens were prepared using ThinPrep® and Papanicolaou staining before being examined microscopically ²⁴. The urine slides were scanned using a Leica Aperio-AT2 scanner at 40× resolution and were stored as 70% quality SVS files representing whole slide images. The slides were manually focused (by a trained technician) on a single plane during scanning, and z-stacking was not used. Patient and slide-level characteristics from the slide-level training and validation cohorts can be found in Table 1. All slides were assessed by a group of five cytopathologists using TPS criteria (negative for high grade urothelial carcinoma, atypical urothelial cells, suspicious for high grade urothelial carcinoma, positive for high grade urothelial carcinoma) ¹².

Methods Overview

In this section, we summarize improvements introduced in AutoParis-X, which will be elaborated on in following sections. AutoParis-X was written using the Python programming language and neural networks were implemented using the PyTorch and Detectron2 frameworks ^{50, 51}. Statistical and machine learning models were implemented in Python and R ^52–54. A graphical overview is provided in Figure 1:

1. Slide processing– Connected components analysis to isolate individual cells and cell clusters, sped up through parallel processing ⁵⁵.

2. Cell border detection (BorderDet)– Isolates cells within urothelial clusters with overlapping cytoplasmic borders through neural network detection model ⁴⁴.

3. Cell-Level Measures:

   1. Morphometric measures– Additional morphological features to improve cell-type classification and atypia estimation (e.g., size / area).

   2. Urothelial Classifier (UroNet)– Used to filter urothelial cells from potentially conflated cell types through a convolutional neural network, which operates on images of cells and their morphometric measures ⁵⁶– trained on an expanded dataset with more cell classes.

   3. NC ratio estimation (UroSeg)– Estimates the NC ratio by neural network pixel-wise segmentation of background, nucleus and cytoplasm. Used as objective marker of atypia.

   4. Atypia score (AtyNet)– For predicted urothelial cells at a particular cutoff threshold, a subjective score which incorporates multiple screening criteria (e.g., hyperchromasia, etc.) is determined using another convolutional neural network which operates on images of cells and their morphometric measures and outputs an atypia score ⁴⁸.

4. Cell- and Cluster-Slide-level scores– Established through a combination of the above scoring methods, counting the number of cells/clusters in the slide with atypical morphology / cluster architecture as defined by previous works ^{43, 48}. Optimal decision cutoffs for determining cellular/cluster atypia were decided using Bayesian Optimization techniques ⁵⁷.

5. Classifier development– Machine learning classifier which integrates cell and cluster level scores and other demographic/specimen characteristics into an Atypia Burden Score (ABS), accounting for repeat measures by patient ^58–64.

6. Model interpretation– A hierarchical logistic regression model was constructed from the machine learning model to identify important indicators of atypia, in addition to analogous univariable models. Helpful graphical displays were generated through an interactive web application ⁶⁵.

7. Demo– A demo was deployed to an Amazon Web Services (AWS) server and software released through GitHub and PyPI.

• Download figure
• Open in new tab

Figure 1: AutoParis-X specimen processing workflow:

A) Connected component analysis isolates candidate cells and cell clusters; B) Individual cells and cytoplasmic borders isolated from cell clusters using BorderDet; C) UroNet isolates specific cell types across slide, in order: i) urothelial cells, ii) polyomavirus infected cells, iii) crystals, debris, RBCs, iv) leukocytes, v) leukocyte clusters, vi) squamous cells; D) AtyNet estimates atypia score for each urothelial cell; E) UroSeg calculates the NC ratio for each urothelial cell after being isolated using the connected component analysis or BorderDet; F) example rich information frame cell and cluster level scores, which cross tabulate statistics across the slide; G) mixed effects machine learning method predicts atypical burden score which correlates with the reported diagnosis; H) cytopathologists can rapidly assess the specimen using the AutoParis-X web application

Slide Preprocessing

As detailed in a previous work, individual objects in the image were identified through a connected component analysis ⁴⁸. In brief, WSI were converted into grey scale images using opencv2 in Python (version 3.8) ⁶⁶. The background of WSIs were converted to white through intensity thresholding of the grey scale image to form an object mask. Small objects, defined as a pixelwise area of 50 or below, were filtered using the remove_small_objects (scipy, Python v3.8) morphological operation ⁶⁷. Large objects (e.g., ink markings) were similarly filtered as defined by a minimal area of 500,000 pixels. After small and large object removal, holes within the object mask were filled through the fill_voids function (which is faster than offerings from the scipy package) ⁶⁸. We leveraged the cupy package (Python v3.8) to reduce compute time through usage of Graphics Processing Units (GPU) where appropriate after extensive timing tests ⁶⁹. Subimages of slide objects (e.g., candidate urothelial cells and clusters) were returned using the scipy regionprops function, which also returned various other morphometric measures and bounding boxes. Inference time and memory usage for the connected component analysis for object identification was reduced through distributed computing procedures (e.g., Dask), which use optimized parallelization to operate on larger-than-memory arrays. Using multiprocessing through dask, operations were also parallelized across subregions within the slide ⁵⁵.

Cell Border Identification for Cell Cluster Analysis

To improve detection of individual cells within clusters, we previously developed a cell detection neural network, BorderDet, (using the state-of-the-art Detectron2 framework) to identify: 1) location of cells through estimation of bounding boxes (one box per cell) and 2) identify cell boundaries by separating overlapping cytoplasm from adjacent cells. BorderDet was developed using cell clusters identified from the cluster-level training cohort. In brief, two cytopathologists (LJV and XL) annotated 800 cell cytoplasmic boundaries for squamous cells, inflammatory cells, negative/atypical urothelial cells, and dense regions of overlapping/indistinguishable cell borders (dense region). BorderDet is an object detection neural network that can detect multiple objects/instances (i.e., cells) in a cell cluster image ⁴⁴. It looks for areas in the image that may contain an object and then assigns a score that indicates how likely it is that the region contains an object. The program labels identified objects with the appropriate label (e.g., squamous cell, dense region) and draws a line around the edges of the object (i.e., segmentation mask) to portray the exact boundary, which can overlap with adjacent cells. This allows the program to accurately identify and locate multiple objects in a single cluster. Objects were then filtered using non-max suppression, a technique which ranks overlapping objects, as defined through their intersection over union (IoU), based on their “objectness” score and removes objects with a lower score ⁷⁰.

To reduce the number of objects assessed using BorderDet, a size filter was enforced, assessing candidate cell clusters with a pixelwise area of at least 1800 pixels, determined through a sensitivity analysis and visual inspection. Parallel processing through multithreading and multiprocessing was integrated using dask for rapid evaluation ⁵⁵. Individual cells extracted through the connected component analysis (area between 256 and 1800 pixels) and objects extracted from clusters using their instance segmentation masks were further assessed using single-cell algorithms which report quantitative metrics of atypia (cell-level measures).

In comparison to the density-based clustering approach that validated urothelial clusters using a CNN (Sanghvi et al.), which could lead to many false negative findings (i.e., approach only “screens out” candidate cell clusters), urothelial cell clusters were identified by BorderDet if they contained urothelial cells ⁴³. This approach improves on watershedding (AutoParis v1) and density-clustering (Sanghvi et al.) techniques as these two methods do not precisely identify cells within larger candidate clusters ^{20, 43, 44, 48}. BorderDet also improves upon previous methods by locating dense urothelial cell architectures with overlapping indistinguishable cytoplasmic borders which are challenging to assess for individual cells. Furthermore, while presence of a dense architectural region in a cluster as defined by an area cutoff was used as an atypia predictor, dense architectures themselves were further subclassified as atypical if surrounding urothelial cells were labeled as atypical (as defined by morphology).

Cellular Morphometric Measures

Various morphometric features were estimated from individual candidate cells, including: 1) area; 2) convex area; 3) eccentricity; 4) equivalent diameter; 5) extent; 6) Feret’s diameter; 7) maximum diameter; 8) filled area; 9) major axis length; 10) minor axis length; 11) perimeter; and 12) solidity, extracted using scikit-image (Python v3.8) ^{56, 71}. These morphometric features were primarily used to help demarcate urothelial cells. As an example, urothelial cells are significantly larger than leukocytes, so cell area is an important criterion for separating the two cell types. Morphometric features were standardized using quantile transformation (implemented in scikit-learn, Python v3.8) within the training set to reduce the influence of any given cell on specifically which morphometric features were important for the assessment ⁷². This places greater emphasis on the imaging findings as means to delineate between different cell types.

Urothelial Cell Classification

Urothelial cell classification was accomplished using UroNet, which was modified significantly from its original incarnation. While AutoParis estimated both the presence and atypia of the urothelial cell simultaneously ⁴⁸, as differentiated from several other specimen constituents, AutoParis-X is chiefly focused on delineating urothelial cells from potentially conflated cell types and slide objects prior to estimating atypia. When aiming to validate the AutoParis algorithm, we noticed that a nontrivial number of urothelial cells lacked a nucleus, potentially related to being out of focus (no Z-stacking) ⁷³, but were not included in our original training set and thus were often confused with other cell types with a smaller nuclear area (e.g., squamous cells). We also identified rare urothelial cells with changes consistent with a Polyomavirus cytopathic effect ^{49, 74}. These cells are benign but assessment can often mimic HGUC and would certainly mislead any attempt to accurately predict the NC ratio and are thus removed by UroNet.

A total of 108,388 and 27,097 cells were manually labeled by two cytopathologists (LJV and XL) and used to train and validate the cell level model respectively from the cell-level training and validation cohort. A breakdown of cell types present in this training and validation cohort is listed in Table 2. These cell images were combined into the following classes: 1) urothelial cells (benign/atypical), 2) urothelial cells with polyomavirus cytopathic effect, 3) debris, crystals and red blood cells (RBC), 4) leukocytes, 5) clusters of leukocytes, and 6) squamous cells. UroNet was developed using a residual neural network (ResNet18), augmented with an auxiliary layer which combines the morphometric information (e.g., area/size, eccentricity, etc.) with features extracted from ResNet18 by fusing the penultimate layer of the network with this information. The auxiliary neural network first maps the number of morphometric features, , to the number of ResNet18 features using a multi-layer perceptron, f_ϕ. Then the morphometric information (same dimensionality as the ResNet features) is fused with the deep learning features using a gated attention operation, which decides dynamically on a cell-by-cell basis which set of features (deep learning, , vs morphometric, to weight more. The weight is dynamically determined using the gating neural network, f_θ⁷⁵.

This operation permits UroNet to filter out cells with significant size differences (e.g., leukocytes are much smaller than urothelial cells). After model training using the PathflowAI package ⁷⁶, the performance of UroNet was assessed using the cell-level validation set through the area under the receiver operating characteristic curve (AUC), reported for each class. To assess how much weight was placed on the morphometric features for prediction, we investigated the attention weights, α, across the validation set. We used Integrated Gradients ^{77, 78}, a deep learning interpretation method, to assess which specific image/deep learning and morphometric features were important for each cell type.

NC Ratio Estimation

For cells classified as urothelial, the NC ratio was calculated for both isolated and cluster cells using a segmentation neural network, UroSeg, which employed a U-Net architecture to assign on a pixelwise basis the presence of nucleus, cytoplasm, or background ^{48, 79, 80}. These areas were annotated/outlined by cytopathologists and UroSeg was trained and validated on 3,690 and 1,231 urothelial cells respectively. Performance was reported using the area under the receiver operating characteristic curve (AUC), reported on a pixelwise basis. For select cell clusters, we compared the impact of running BorderDet, followed by UroNet and UroSeg to calculate the NC ratio as compared to running UroSeg then watershedding, as was originally done by the previous AutoParis algorithm.

Atypia Score

Several cytopathologists determined whether every urothelial cell extracted from the cell-level training and validation cohort (Table 2) was benign or atypical, based on existing markers of atypia (e.g., presence of nuclear membrane irregularity, abnormal chromatin, hyperchromasia, etc.). From this information, AtyNet, a CNN based on ResNet18 with a similar architecture as UroNet, was trained to recapitulate these subjective findings ⁸¹. For every urothelial cell, AtyNet calculates a subjective marker of atypia– the atypia score– which is a value from 0-1 that reflects the probability that a cell is atypical. We used IntegratedGradients, a deep learning interpretation method, to assess which specific image/deep learning and morphometric features were important for atypia assignment.

Calculation of Cell and Cluster Slide-Level Scores

All extracted individual cell and cluster level statistics are placed into Rich Information Frames (RIF), which are data frame/tabular data structures ⁴⁸. For any given WSI, there are three RIFs (see Table 3 for description of features):

1. Isolated-Cell-RIF: Stores morphometric measures; bounding box locations within specimens, cell type assignment probabilities; NC ratios; and atypia scores for each cell not associated with clusters (isolated urothelial cells).

2. Cluster-Cell-RIF: Stores morphometric measures; bounding box locations within specimens; cell type assignment probabilities; NC ratios; and atypia scores for each cell associated with clusters, in addition to their cluster assignment label (cluster urothelial cells).

3. Cluster-RIF: Stores bounding box locations within WSI; cluster size; cytoplasmic borders; area of dense regions in cluster; and associated cluster label/identifier. Information on cellular atypia (e.g., number of atypical cells), number of urothelial cells, amongst other cluster-level measures, were added to this RIF from the Cluster-Cell-RIFs.

All RIFs are cross-tabulated to form a Slide Inference Frame (SIF), which represents slide-level statistics, aggregated across all urothelial cells and urothelial cell clusters. This is accomplished by thresholding the cutoff probabilities for the cell and cluster-level scores and counting the number of cells and clusters which meet these criteria. For instance, given an atypia score cutoff of 0.7 (i.e., cell is atypical if AtyNet assigns a 70% probability), a cluster is deemed to exhibit cellular atypia if, for instance, more than 20% of the cells within the cluster are atypical under this definition. Based on the definition of a urothelial cluster (e.g., number of urothelial cells), the number of atypical clusters within the WSI can be estimated. All urothelial cells with an NC ratio of 0 were removed prior to calculating these scores. SIF contains the following statistics:

1. Isolated cell subscores: Derived from Isolated-Cell-RIF, for cells which were not associated with clusters, including the following statistics: 1) number of urothelial cells;

2. number of atypical urothelial cells as determined using the atypia score; 3) number of atypical urothelial cells as determined using the NC ratio; 4) number of urothelial cells; and 5) center and spread of various morphometric measures.

3. Cluster cell subscores: Derived from Cluster-Cell-RIF. Similar to isolated cell subscores, only considering cells which were associated with / identified within clusters.

4. All cell subscores: Combines isolated and cluster cell subscores, considering all cells, irrespective of whether there was a cluster assignment.

5. Cluster subscores, representing aggregate Cluster-RIF statistics, including: 1) number of urothelial clusters (defined by a minimum threshold of urothelial cells); 2) number of atypical urothelial clusters (defined by either NC ratio or atypia score); 3) number of dense clusters; and 4) number urothelial clusters that are both atypical and dense. Unlike the previous three scores which focus on individual urothelial cells, identified urothelial cell clusters represent the principal unit of analysis.

Using AutoParis-X, RIF-SIF scores were calculated across the slide-level training and validation cohorts. We added the following patient-level characteristics to the RIF-SIF scores: 1) age; 2) sex; 3) history of hematuria; and 4) specimen source ^{82, 83}. We also noted where slides contained significant blood, high cellularity, acellularity, neobladders (abundant degenerated enterocytes) and scanning artifacts.

Estimating Specimen Atypia with Machine Learning

Specimen atypia was reported through dichotomization of TPS categories into the following classes: 1) negative, atypical and 2) suspicious, positive. The Atypia Burden Score (ABS) reflects the predicted probability of a specimen being atypical as assessed by AutoParis-X. We implemented several machine learning and statistical modeling approaches to predict specimen atypia, including: 1) generalized linear mixed effects modeling (hierarchical logistic regression; GLMM; brms package, R v4.1), accounting for patient- and pathologist-level random intercepts, 2) Random Forest, which does not account for clustering by patient, 3) Gaussian Process Tree Boosting (GPBoost), and 4) Bayesian Additive Regression Trees (BART) ^{58–61, 64}. GPBoost and BART account for clustering by patient by fitting patient- and pathologist-level random intercepts while capturing interactions and nonlinear associations between SIF predictors using ensemble tree models, f_θ(x̄):

Overall model performance was communicated using fivefold cross-validation, which randomly partitions the data into a training and validation set and reports the overall performance (using the AUC) over the validation folds. Specimens belonging to the same patient were partitioned into the same training/validation fold for each cross-validation split to avoid potential inflation of test statistics. Confidence intervals (CI) were reported using 1000-sample nonparametric bootstrapping of each fold to yield 1000 samples of cross-validation statistics. Cell and cluster-level thresholds (e.g., atypical cell if NC>0.7; atypical cluster if at least 3 urothelial cells are atypical), which are used to generate RIF-SIF scores, were optimally aligned with specimen atypia through a Bayesian Optimization routine ⁵⁷.

Interpretation

We identified significant ABS predictors by extracting salient interactions from the tree ensemble models and reporting odds ratios (OR) from univariable and multivariable Bayesian GLMM models: . As many of the ABS predictors were highly multicollinear, variance inflation factors and horseshoe lasso priors were used to select predictors ^{84, 85}. Univariable associations adjusting for age, sex and hematuria were reported to give credence to omitted collinear predictors in the multivariable statistical modeling.

Hierarchical Bayesian cumulative link models (i.e., ordinal regression) in a similar specification were also used to report associations between the predictors and specimen atypia, treating the urine cytology assignment as an ordinal variable ^{86, 87}. Statistical significance was reported using the p-value, as derived from the probability of direction (pd): p ≈ 2 ∗ (1 − pd). A p-value less than 0.05 indicates a significant atypia predictor. Credible intervals, similar to confidence intervals, communicated uncertainty in the effect estimates.

Web Application and Software Availability

We also developed an interactive web application which allows for rapid assessment of cytology slides. In brief, users first select a slide to examine. An ABS score is returned for the specimen as assessed using AutoParis-X. The Cell-RIF is converted into a 2D scatter plot of the NC ratio and atypia score– each point represents a cell. Using a “lasso tool”, users select cells within this scatterplot. The urothelial cells are highlighted on a zoomable WSI viewer (openseadragon) and additionally made available through an image gallery for additional examination (Figure 2) ⁸⁸. The WSI viewer will highlight cells based on their relative degree of atypia as assessed algorithmically, focusing the end-user on a small subset of potentially malignant cells. A demo of this interactive web application can be found at the following URL: http://edit.autoparis.demo.levylab.host.dartmouth.edu/ (user: edit_user, password: qdp_2022; full-screen display is encouraged for optimal viewing experience). The web application also features a tutorial video for operating the application. The AutoParis-X software is also open-source, available to download on GitHub (https://github.com/jlevy44/AutoParisX) and installable using the following PyPI package: autoparis. Users aiming to run AutoParis-X will need to train compatible neural networks as neural networks were only trained on data from a single institution and would need additional finetuning to generalize.

• Download figure
• Open in new tab

Figure 2: AutoParis-X Web Application:

A) Cytopathologist selects patient/specimen scanned and processed the previous day, which outputs Atypia Burden Score; B) Urothelial cells are identified based on a cutoff probability selected by the user; C) Individual cells are plotted using scatter plot, which depicts each cell’s NC ratio and atypia score; user selects most atypical cells for viewing via the WSI viewer and gallery using the “Lasso” tool; D) WSI viewer– red points are sized by degree of atypia and identify important urothelial cells to assess/zoom in; E) gallery view enables rapid examination of individual cells, sorting them by their degree of atypia

Performance of UroNet

UroNet demonstrated remarkable performance in the task of delineating among 6 different classes of cell types / objects to determine which cells are urothelial (Figure 2; Table 4). Figure 3A demonstrates a nearly perfect ROC curve (AUC=0.997 macro-averaged) for all 6 cell types across the validation set, indicating high classification accuracy. In addition, raw imaging features interpreted using IntegratedGradients corroborated with known histomorphology for specific cell types (e.g., highlighting dense chromatin to depict urothelial cells, surrounding membrane for squamous cells, etc.; Figure 3B). Many morphometric features were found to be important– for instance: 1) eccentricity as a defining feature of urothelial cells versus other cell types, 2) solidity for RBCs, 3) convex area as an important predictor for leukocyte clusters which have highly irregular formations, and 4) both convex area and solidity for squamous cells, which are larger than the other cell types and typically solid shapes without any notable deformations (Supplementary Figure 1). These findings suggest that UroNet can accurately identifying urothelial cells, important for establishing assessment of urothelial cells as the basis for AutoParis-X’s automated assessment.

• Download figure
• Open in new tab

Figure 3: Performance of UroNet/UroSeg/AtyNet:

A) Receiver operating characteristic curves for each cell type from the internal validation set (UroNet) and for delineating atypical versus benign urothelial cells (AtyNet); B) Integrated Gradients heatmap localizing important features identified using UroNet for urothelial cells, squamous cells and leukocyte clusters; C) Integrated Gradients heatmap localizing important features identified using AtyNet for one benign urothelial cell / cell cluster, followed by two atypical cell images; D) Example ground truth segmentation masks (left; background-black, cytoplasm- red, nucleus- yellow), original images (center) and segmentation masks predicted using UroSeg (right); E) Receiver operating characteristic curves for background, cytoplasm and nucleus (pixelwise assessments) from the internal validation set (UroSeg); F) Ground truth versus UroSeg predicted NC ratios, derived from the segmentation results

Performance of UroSeg

UroSeg, a neural network segmentation tool, demonstrated excellent performance on our internal validation set in predicting the pixelwise presence of the nucleus and cytoplasm (AUC=0.971 macro-averaged) in order to calculate nuclear to cytoplasm (NC) ratio (Figures 2-3; Table 4). Figure 3F also shows nearly perfect receiver operating characteristic curves for both the nucleus and cytoplasm, indicating the high accuracy of UroSeg in predicting these structures. Additionally, we found that the NC ratios calculated from the segmentation masks produced by UroSeg correlated nearly perfectly with the ground truth NC ratios (r=0.965; MAE=0.015) annotated by the cytopathologists (Figure 3G). Figure 3E demonstrates the alignment of the true and predicted nuclear and cytoplasmic segmentation masks, further highlighting the accuracy of UroSeg. UroSeg was similarly effective when used in conjunction with BorderDet, our previously established urothelial cluster border separation tool. Cells extracted from urothelial clusters using BorderDet and confirmed to be urothelial via UroNet were assessed using UroSeg. We compared the NC ratios, averaged across each urothelial cluster, in our internal validation set with what was accomplished using watershedding techniques (which divided the clusters after seeding the watershed based on the location of the nuclei). Watershedding was not sensitive to the cell type as it did not leverage BorderDet and UroNet. In addition, for clusters containing urothelial cells and background debris or other confounding cell types, watershed heavily underestimated the NC ratio (Figure 4). This was universal across all of the urothelial clusters in the internal validation set. Through visual examination, it is clear that by precisely demarcating cytoplasmic borders between immediately adjacent and overlapping cells, BorderDet and UroNet allow for precise estimation of the NC ratio. Opting for alternative assessment approaches (e.g., watershedding) could reduce the predictive capacity of slides containing abundance of urothelial cell clusters by removing or unnecessarily skewing the reported statistics for these cells as compared to isolated cells.

• Download figure
• Open in new tab

Figure 4: Performance of BorderDet and UroSeg on estimating NC ratios for cells in clusters:

A) Estimates derived using watershedding underestimate the NC ratio, whereas detecting the urothelial cytoplasmic borders then using UroSeg (segmentation masks plotted over detected urothelial cells) to estimate the NC ratio leads to a higher and more accurate NC ratio; final cluster contains dense region of significantly overlapping and indistinguishable cytoplasmic borders, dense area used as a predictor for AutoParis-X; B) Scatterplot comparing watershed-derived and BorderDet derived NC ratios; C) Shift plot indicating BorderDet NC ratios are higher than that achieved using watershedding

Performance of AtyNet

Performance for AtyNet, the neural network which provides an atypia score estimate for each urothelial cell, was equally promising (Figure 2; Table 4). The algorithm achieved an area under the receiver operating characteristic curve of 0.917 on the internal validation set, indicating a strong ability to distinguish between atypical and normal cells. Model interpretation using integrated gradients revealed that the algorithm placed a high emphasis on irregularities in the nuclear membrane as a key feature in determining cytological atypia (Figure 2B) ⁵⁶.

ABS Classifier Performance

Individual cell and cluster level features were cross tabulated across the slide and assessed using multiple statistical and machine learning algorithms. Many cellular and cluster level features correlated closely with specimen atypia (Supplementary Figures 2-4). Atypical urothelial cells as defined by both the NC ratio and atypia score, which were contained within clusters were, in some cases, more predictive of specimen atypia than assessment of isolated cells alone (e.g., cells with high NC ratio in clusters were more predictive than isolated cells with high NC ratio), further suggesting the importance of employing BorderDet for separating cells. The number of urothelial cells and cell clusters correlated directly with potential for malignancy. Urothelial cell clusters which were both atypical and contained dense regions were the third most predictive variable when assessed using univariable regression.

As part of the AutoParis-X framework, each machine learning model outputs the Atypia Burden Score (ABS)– the probability of assigning suspicious or positive UC exam as judged using AutoParis-X. Across all algorithms, ABS correlated closely with specimen atypia. The machine learning models which accounted for patient and pathologist-level variation, GPBoost and BART, outperformed all other approaches with AUCs of 0.89 and 0.88 respectively (Figure 5A; Table 5). The generalized linear mixed effects models also performed well. Across all models, ABS scores preserved the ordering of the UC categories (Negative<Atypical<Suspicious<Positive; Figure 5B). We fit an ordinal regression model to this data, which demonstrated a strong positive association with atypia (UC categories; β = 3.61; 95%CI: [3.12 − 4.11]; p < 0.0001). This information is corroborated by density heatmaps depicting the NC Ratio and Atypia score for individual urothelial cells across the entire cohort, after being filtered using UroNet. This yielded more than 6 million cells, which were separated based on their UC class. Figure 5D demonstrates the progression in cellular atypia across the categories– negative cases typically do not contain cells that have both high NC ratio and atypia, while these cells can be increasingly found at higher UC categories. Positive cases contain many cells that are both highly atypical with high NC ratio.

• Download figure
• Open in new tab

Figure 5: ABS Classifier Performance:

A) Receiver operating characteristic curves illustrating performance of ABS classifiers; B) Boxplot of raw ABS scores predicted by GPBoost and Random Forest by UC class; C) Point estimates and 95% credible intervals for predictors uncovered from final multivariable Bayesian hierarchical model; D) Density plot of NC Ratios and Atypia scores cross tabulated across over 6 million cells from the retrospective cohort, divided by UC classes, demonstrating progression of cells to take on higher NC ratios and Atypia scores at higher UC classes

Univariable and Multivariable associations with Specimen Atypia

Table 6 demonstrates the importance of the individual slide level predictors through both univariable and multivariable regression modeling. A few predictors remained in the unpenalized statistical model after applying the horseshoe lasso (Figure 5C). This included positive associations with number of clusters, number of both atypical and dense clusters, number of isolated atypical cells and an interaction between age and atypia. The interaction demonstrates that overall specimen atypia younger individuals more greatly impacted by number of atypical urothelial cells as compared to older individuals.

Web Application Example

As a demonstration of Autoparis-X’s ability to facilitate rapid examination of UC specimens, we examined four specimens with the web application (see Supplementary Figures 5-7 for screenshots). Among thousands of specimens examined using this web tool, select cases (negative, atypical, suspicious, positive) can be further inspected using the demo application (see Web Application and Software Availability). The first case (Supplementary Figure 5) yielded an Atypia Burden Score of 0.14. Urothelial cells were selected with high atypia and were plotted on the WSI, revealing their locations. Zooming in on the WSI confirmed the reported cell-level statistics. We also used the table as means to rapidly examine all atypical cells in order of decreasing atypia as a faster method to examine cells versus zooming in using the web application. These examinations confirmed that this was in fact an atypical specimen. The second case produced an atypia burden score of 0.6– a similar examination revealed specimen atypia on par with that of a suspicious assignment. The final case was a positive patient with an atypia burden score of 0.76. We focused on only a few cells which demonstrated the highest potential for malignancy in order to focus our examination given the high cellularity of the specimen. Many of these cells were nested in urothelial cell clusters. This search identified cells which were indeed highly malignant morphologically, allowing for rapid assignment of a positive finding. In Supplementary Figure 8, we used the WSI viewer to zoom in on a few malignant cells identified using the AutoParis-X web application.