Abstract
Background Tissue segmentation of histology whole-slide images (WSI) remains a critical task in automated digital pathology workflows for both accurate disease diagnosis and deep phenotyping for research purposes. This is especially challenging when the tissue structure of biospecimens is relatively porous and heterogeneous, such as for atherosclerotic plaques.
Methods In this study, we developed a unique approach called EntropyMasker based on image entropy to tackle the fore- and background segmentation (masking) task in histology WSI. We evaluated our method on 97 high-resolution WSI of human carotid atherosclerotic plaques in the Athero-Express Biobank Study, constituting hematoxylin and eosin (H&E) and 8 other staining types.
Results and Conclusion Using multiple benchmarking metrics, we compared our method with four widely used segmentation methods: Otsu’s method, Adaptive mean, Adaptive Gaussian and slideMask and observed that our method had the highest sensitivity and Jaccard similarity index. We envision EntropyMasker to fill an important gap in WSI preprocessing and deep learning image analysis pipelines and enable disease phenotyping beyond the field of atherosclerosis.
Introduction
Atherosclerosis is a chronic inflammatory process resulting in arterial stiffening and plaque formation, and is the leading cause of myocardial infarction, ischemic stroke, and peripheral artery disease1,2. Historically, researchers and pathologists have characterized atherosclerotic plaque through standard histology and light microscopy analysis3–6. The composition of atherosclerotic plaques is highly variable, with different plaque types having distinct clinical manifestations3,7. For instance, more stable, fibrous-rich atheroma plaques are typically asymptomatic, whereas unstable, thin-cap fibroatheroma plaques are more prone to rupture and thrombus formation underlying cerebral or coronary events5,6. However, the value of atherosclerotic plaque composition in predicting cardiovascular events remains a subject of debate and ongoing research7–9.
While histology-based assessment of cancer and other diseases is often required for clinical diagnosis, histological analysis of atherosclerosis can reveal the extent of disease progression as well as underlying etiology. For example, in pre-clinical models of atherosclerosis, the amount of smooth muscle content and collagen extracellular matrix in the plaque often correlates with greater plaque stability8. These features can now be easily captured from stained tissue sections and digitized as high-resolution Whole Slide Images (WSI). WSI data provides a rich resource for quantitative and qualitative image analysis and has been a focus of digital pathology10. Overall, this has simplified archiving, enabled remote diagnosis, and accelerated both clinical decision-making and research investigations11.
Application of quantitative WSI analysis pipelines requires access to uniformly processed and well-characterized tissue biobanks. The Athero-Express study3 is a large-scale vascular tissue biobank comprising over 3,600 carotid and femoral endarterectomy surgical specimens, which include detailed clinical outcomes and follow-up. This has led to several prospective studies to correlate local atherosclerotic plaque composition with future local and systemic vascular outcomes, using histology7–10, RNA12,13, genetics14, and protein15,16 data. In the histology-based analyses, local plaque indices such as plaque hemorrhage and neovascularity were shown to correlate with more adverse vascular outcomes7,9.
Accurate tissue segmentation is a necessary initial step in WSI analysis in digital pathology workflows, particularly for deep learning-based classification. However, the process of WSI acquisition has its own technical challenges. Whole-slide scanners are known to fail to identify or “mask” the entire tissue region, e.g., due to dust specs, transparent tissue types (e.g., adipose) or weak immunostaining results. Omitted tissue areas could pose a significant problem for diagnostic accuracy, e.g., such as in cancer, identifying malignant regions in lymph nodes17. These tissue recognition mistakes may have costly downstream consequences in automated digital pathology workflows, be it for diagnostic or research purposes. By highlighting the tissue-specific area, deep learning algorithms can run more efficiently by only analyzing the informative portions of the WSI. For example, only 22 percent of the total WSI area was estimated in the CAMELYON17 challenge dataset18. In addition, inter- and intra-image color fluctuations, staining and slide preparation aberrations must also be considered through a robust tissue segmentation method to improve WSI interpretation.
Masking in histological sections
The most straightforward method for identifying tissue on a white background is to set a specified threshold for the grayscale version of the image19, specific color channels20, or the optical density of the red, green, and blue (RGB) color channels21. Alternatively, WSIs can be split into a uniform image patch grid, then grayscale image thresholded, and the resultant binary tissue map used to reject image patches with insufficient tissue 22,23.
The most commonly used tissue masking method is Otsu’s adaptive thresholding 24, a technique applied to WSI based algorithm training in the CAMELYON17 challenge and other studies18,25. Otsu’s method has been extended by capturing additional thresholding areas from the edge of the binary mask26. In addition, several methods have been developed to improve WSI based tissue recognition. For instance, GrabCut27 for image segmentation of regions of interest, Watershed for cell or nuclei segmentation,28 and foreground extraction from structural information (FESI)29 for fore-and background segmentation. Thresholding at a fixed value and Otsu’s approach are easy to implement but have been difficult to generalize to more variable stains or clinical specimens. A more accurate approach is still needed in digital pathology workflows, especially for specimens containing small empty spaces between extracellular matrix fibers (e.g., collagen) or due to decalcifying procedures, and transparent tissue for example in adipose tissue or fat deposits in atherosclerotic plaques.
In this paper, we propose a fully automated approach for separating foreground (tissue) and background in histopathological bright-field microscopy images of hematoxylin and eosin (H&E)-stained samples. Our method, EntropyMasker, is unaffected by changes in scanning or image processing conditions, by using a measure of local entropy and generating corresponding binary tissue masks.
Methods
Our method: EntropyMasker
For uniform image processing, ideally the slide background is evenly achromatic. Unfortunately, in practice, there is often inconsistency of the background color due to variations in image capturing color temperature (e.g., from shades of blue to yellow), resulting in edge artifacts from tile-based scanning methods30. We corrected this background-color inconsistency by applying color hysteresis thresholding based on color distance from the “most common” background color. By doing so, we were able to easily determine the “seed regions” of the slide background (i.e., color range unique to the background).
In information theory31, entropy is defined as the log-base-2 of the number of possible outcomes for a sent/received message. In image analysis, entropy refers to the degree of randomness or complexity of pixels in a defined region or neighborhood of pixels. We defined the local entropy of a specific region as follows:
where pF (i) represents the probability of a grayscale pixel, i, in a local footprint region.
In the case of pF (i) = 0 for some i, the corresponding sum value and 0log20 is set to 0, which is consistent with the limit: 
.
Based on this approach, informative pixels have a low probability and contribute less to the local entropy, whereas noise pixels have a high probability and contribute more to the local entropy. Also, the background of an image has lower entropy due to higher brightness and more homogeneous texture, whereas foreground tissue has higher entropy due to the diversity of pixel intensities in the local neighborhood.
To extract tissue from the background of the image, we applied a filter to the generated local entropy value map. The filter is a threshold for determining the optimal range of local entropy values to distinguish background noise from tissue in the WSI. Since the background of the image has a lower entropy value, we set the threshold as the local minima of the histogram of the entropy map. We then assigned a Boolean value to each pixel in the WSI, defined as background or not, based on the entropy values compared to the threshold. These binary values were used to create the tissue segmentation mask for each WSI.
Otsu’s method
Otsu’s method is a clustering-based image thresholding algorithm24, commonly utilized in image analysis and digital histopathology applications25,30,32,33,34. Following a bimodal histogram, the technique assumes that the image has two classes of pixels (tissue pixels and background pixels). It determines the best threshold for dividing the two groups such that their total intra-class variance is as low as possible.
Adaptive thresholding
Adaptive thresholding is another widely used method for calculating the threshold value for smaller areas of WSIs35,36,37,38. Typically, these threshold values are calculated in two different ways, either using the mean of the neighborhood area, or a Gaussian weighted sum of neighborhood values.
slideMask method from slideToolKit
The slideToolKit39 is an assistive tool set for the histological quantification of whole-slide images (https://github.com/swvanderlaan/slideToolKit). The slideMask tool in slideToolKit uses convert from ImageMagick (https://imagemagick.org) and a miniature version of the WSI to generate a mask (https://github.com/swvanderlaan/slideToolKit/blob/master/slideMask). The image is first blurred to eliminate dust and speckles. Then Fuzz in ImageMagick40 (which is computed as the root mean squared difference between two colors of the pixels) is used to create a fuzzy, non-stringent selection to identify the white background, which is subsequently substituted with black. The slideMask tool has settings for blur and fuzziness that may be explored and altered. Manual adjustments may be made to generated masks in any image editor (such as free GNU Image Manipulation Program; GIMP38,41). Unwanted regions on the WSI (such as marker stripes or air bubbles beneath the coverslip) may necessitate this procedure.
All proposed methods were implemented in the Python programming language using OpenCV42 and other required packages (see Data and code availability for more details).
Athero-Express Biobank Study
Patient population
Atherosclerotic plaques were obtained from patients undergoing a arterial endarterectomy procedure and included in the Athero-Express Biobank Study (AE), an ongoing biobank study at the University Medical Centre Utrecht (Utrecht, The Netherlands) and the St. Antonius Hospital (Nieuwegein, The Netherlands)3. The medical ethical committees of the respective hospitals approved the AE which was registered under number TME/C-01.18. This study complies with the Declaration of Helsinki, and all participants provided informed consent. However, considering national laws plaque-material are considered ‘waste biomaterial’ and are always allowed to be used without any personal information regardless of informed consent. In this study we also considered this ‘waste biomaterial’, hence no clinical information, e.g., age, is given here nor when sharing data of the relevant samples (n=3). The study design of the AE was described before3, but in brief: during surgery blood and plaques are obtained, stored at -80°C and plaque material is routinely used for histological analysis3,43.
Whole-slide staining and scanning
The standardized (immuno)histochemical analysis protocols used in the AE biobank have been described previously.3,43 In short, 4-micron cross sections of the paraffin-embedded segments were cut using a microtome, and 8 different stains were applied for endothelial cells (CD34), macrophages (CD68), elastic Van Gieson (EvG), fibrin, red blood cells (glycophorin C, GLYCC), hematoxylin and eosin (H&E) for nuclei, collagen (picrosirius red, SR), and smooth muscle cells (SMCs, α--actin) on consecutive slides. We set up ExpressScan to obtain whole-slide images (WSIs) by scanning stained slides at 40x using a Roche Ventana iScan HT (https://diagnostics.roche.com/global/en/products/instruments/ventana-iscan-ht.html) or Hamamatsu C12000-22 Digital slide scanner (https://www.hamamatsu.com/eu/en/product/life-science-and-medical-systems/digital-slide-scanner/index.html). WSIs were stored digitally as z-stacked .TIF (Roche) at 0.25 micron/pixel or .ndpi (Hamamatsu) at 0.23 micron/pixel brightfield microscopy images10. The slides numbers used for this project are listed in Supplemental Table 1.
Manual tissue annotation and masking
The ground truth tissue areas for each WSI were annotated manually at 40x magnification using QuPath44. If the tissue had disintegrated into several disjointed fragments during preparation or staining the annotators marked them with a single enclosing polygon. Binary masks of these tissue annotations were then generated using groovy scripts in QuPath.
Statistical analyses
We assessed the algorithms using several metrics: The Jaccard index, sensitivity, False positive Rate (FPR) and pixel accuracy, comparing the output of the algorithms to the ground truth binary reference masks of the 97 images in the test set.
Jaccard index was applied to assess the performance of our masking algorithms, by measuring the overlap area of target masks with the ground truth masks annotated by experts and masks of the algorithm outputs divided by the area of the union of both masks. Mathematically, sensitivity, false positive rate and pixel accuracy can be expressed as:
Data and code availability
The histological data as used here are available through https://doi.org/10.34894/GN4YOS.
All documented code and tutorial to run EntropyMasker can be found here https://github.com/CirculatoryHealth/EntropyMasker.
Ethics statement
The studies involving human participants were reviewed and approved by Medisch Ethische Toetsingscommissie (METC) Utrecht. The patients/participants provided their written informed consent to participate in this study.
Evaluation and results
We developed EntropyMasker, a novel masking method using entropy-filtering (Figure 1). First, we compared our method against three conventional thresholding methods: Otsu’s, adaptive thresholding using the mean of neighborhood area as the threshold, and adaptive thresholding using the Gaussian weighted sum of neighborhood values as the threshold. We also evaluated these methods against the previously developed slideMask tissue masking method in the slideToolKit39.
Atherosclerotic plaque tissue samples were collected from arterial endarterectomy surgeries, sectioned, and mounted on tissue slides as part of the Athero-Express Biobank Study. The slides were subjected to histological staining for overall morphology (hematoxylin & eosin; H&E), collagen deposition (Picrosirius red; SR), fibrin deposition (Fibrin), elastin fibers (Elastica Van Gieson; EVG) or immunohistochemical staining for cell-type specific markers: alpha-smooth muscle actin (a-SMA), macrophages (CD68), endothelial cells (CD34), or erythrocytes (glycophorin C). Whole-slide images of each slide were generated using high-resolution scanners and images were stored in a database (∼25,000 images). Red, Green, Blue (RGB) color images were converted to grayscale images and grayscale pixels were used to calculate entropy threshold map for histogram analysis, and tissue segmentation was performed to create tissue mask of WSIs for subsequent data analysis and benchmarking. Schematic was created with BioRender.com
We evaluated each method using 97 WSIs (Supplemental Table 1 for details on usage and Supplemental Table 2 for a minimalistic baseline table) at 40X magnification on layer 6 (pixel spacing at layer 6 is around 8.36µm) consisting of 8 different stains: H&E, picrosirius red, Fibrin, EVG, a-SMA, CD34, CD68, and GLYCC. We did not apply our method to layer 1 due to memory constraints. The average Jaccard index, also referred to as the Intersection over Union (IoU) metric, of the adaptive thresholding method where its threshold value is the weighted sum of neighborhood values where weights are a gaussian window was 0.7702 and the average sensitivity was 0.8597. This masking method produced the highest pixel accuracy among all of the traditional methods which was 0.9485. Otsu’s method had a much lower false positive rate than any of the approaches examined at the cost of much lower average Dice scores and sensitivity levels. The quality of the outcome of the slideMask algorithm was unevenly distributed resulting in a relatively low Jaccard index (IoU) across all the tested methods. The average false positive rate was 0.0153, which is the lowest of all masking methods in comparison, and its average pixel accuracy was 0.9133.
Our automatic local entropy-based masking method, EntropyMasker, had the best average Jaccard index (IoU) 0.8837 and the highest sensitivity 0.9685, outperforming Otsu’s method, two adaptive methods and slideMask masking method in both metrics. The average false positive rate was 0.0176 which is much lower than Otsu’s method, and two adaptive methods. Table 1 contains the complete set of metric values and Figure 2 shows the summary statistics of the Jaccard index across all 97 slides. The results of comparing the 5 masking methods for an H&E-stained whole-slide image are shown in Figure 3 where generated tissue masks are overlaid by pseudocolor (green).
The method with the highest evaluation metric is shown in bold. Metric values represent mean ± SD.
Comparison of EntropyMasker with other masking methods, Otsu, slideMask, and adaptive thresholding methods, using manually annotated tissue sections as ground truth and calculated Jaccard index (intersection over union metric). 97 images were used from 65 individual patient samples to evaluate each method. Boxes represent the median and interquartile range (IQR) and whiskers represent up to 1.5X IQR. Outliers (greater than 1.5X IQR) are shown as diamonds.
A) Expert generated annotation (ground truth) using Qupath Software (Version 0.3.2)44. B) Our EntropyMasker method. C) Otsu’s method. D) Adaptive method where the threshold value is the mean of neighborhood area. E) Adaptive method where the threshold value is the weighted sum of neighborhood values where weights are a gaussian window. F) slideMask method. All the generated tissue masks are overlaid by pseudocolor (green).
To demonstrate the generalizability of our automatic entropy method, we tested on whole-slide images with 8 different types of staining. Examples of masking results of our proposed method are shown on Figure 4 where generated tissue masks are overlaid by pseudocolor (green).
Images of the upper part are the original whole slide images at layer 6 with 8 different types of staining methods respectively and images of the lower part are the examples of masking results of our EntropyMasker method where generated tissue masks are overlaid with pseudocolor (green). (A) H&E; (B) Glycophorin C; (C) Sirius red; (D) Fibrin; (E) CD68; (F) CD34; (G) Smooth muscle actin; (H) EVG.
The computation speed for Otsu’s method, adaptive methods, slideMask and proposed EntropyMasker method on a 3.6GHz Quad-Core Intel Core i7 processor, 32GB 2400MHz DDR4 memory iMac are 0.08, 0.11, 0.25 and 0.34 seconds per one million pixels, respectively.
Conclusions and discussion
Overall, our automatic entropy masking method, EntropyMasker, performed well on atherosclerotic plaque cross-sectional WSIs derived from 8 different types of stains. With a relatively high average Jaccard index, our method was able to separate out the foreground and background accurately and consistently from these complex images (Figure 2 and 3).
In comparison, we observed the popular masking method, Otsu’s, and our previous method, slideMask, both tended to miss some tissue areas along the tissue boundary. This likely results from the relatively lower intensity of these areas, where Otsu’s method sets a threshold in the middle of two peaks, thus partially omitting the higher intensity pixels compared to the threshold (Figure 3). The slideMask method computes the root mean squared difference between two colors, however this becomes problematic when the intensity of tissue area pixels is relatively large (Figure 3).
We also found that the adaptive methods tend to produce ‘porosity’ during the masking, since these methods miss small regions of the image which are supposed to be regarded as tissue (Figure 3). These large-scale missing tissue makes Otsu’s, slideMask and adaptive thresholding methods impractical for use in many image processing workflows10,39, especially when applied to atherosclerotic plaque images. Given the limited availability and heterogeneous nature of these plaques, an ideal masking method will retain all the plaque components, which can be used to discover tissue or compartment-specific markers for disease progression and phenotyping. This is also a valuable feature of EntropyMasker, which can be implemented at the preprocessing stage of various deep learning pipelines for other diseases and research areas which require a complete WSI masking step45–47.
Despite the accurate and consistent performance of EntropyMasker, it is worth pointing out some known limitations. In comparison to the other methods that we evaluated, our method has the highest sensitivity, and thus tends to over-include a few pixels outside the edges of tissue (Figure 4). While these regions represent a small fraction of the total tissue areas that we tested, this should be more carefully considered when evaluating smaller tissue sections11. Also, our method had slightly lower pixel accuracy than the adaptive methods, which may be important for tasks involving more subtle intensity differences in defining target regions of the tissue48. Compared to the other methods discussed, ours is slightly slower (0.34 seconds per one million pixels compared to 0.11 seconds for Otsu’s) which may pose an issue when examining thousands of images, although this should be negligible since our method is scalable on any high-performance compute cluster.
In conclusion, we demonstrated the effectiveness of our proposed method for tissue masking on human atherosclerotic plaque WSIs of different types of stains including H&E, CD34, CD68, EVG, smooth muscle cell α-actin, picrosirius red, fibrin and Glycophorin C. By evaluating our method against other popular masking methods and those recently developed, we also demonstrated that our entropy-based masking method is scalable and had optimal performance across these WSIs. Thus, we envision EntropyMasker to serve an important role in WSI preprocessing and deep learning image analysis pipelines and enable disease phenotyping beyond the field of atherosclerosis.
Data Availability
The histological data as used here are available through https://doi.org/10.34894/GN4YOS. All documented code and tutorial to run EntropyMasker can be found here https://github.com/CirculatoryHealth/EntropyMasker.
Author contributions
We used the CRediT taxonomy (https://journals.plos.org/plosgenetics/s/authorship#loc-author-contributions) to determine the author’s contributions summarized below.
Funding
Funding for this research was provided by National Institutes of Health (NIH) grant nos. R00HL125912 and R01HL14823 (to CLM), and a Leducq Foundation Transatlantic Network of Excellence (‘PlaqOmics’) grant no. 18CVD02 (to CLM and SWvdL), and EU H2020 TO_AITION grant no. 848146 (to SWvdL).
Disclosures
Dr. Sander W. van der Laan has received Roche funding for unrelated work. Dr Craig A. Glastonbury has stock options in BenevolentAI and is a paid consultant for BenevolentAI, unrelated to this work. Dr. Clint L. Miller has received AstraZeneca funding for unrelated work.
Supplemental Material
Supplemental Tables
Some slides were also used for figures. The specific stains used include: CD68 (macrophages), α-smooth muscle actin (a-SMA) (smooth muscle cells), CD66b (neutrophils), CD34 (endothelial cells), glycophorin C (GLYCC) (red blood cells), hematoxylin and eosin (H&E) (nuclei), picrosirius red (SR) (collagen), fibrin (FIBRIN), elastic Van Gieson (EvG). WSI are obtained at 40x magnification, and stored as z-stacked .TIF images as produced by the Roche Ventana iScan HT with 0.25 micron/pixel or .ndpi images as produced by the Hamamatsu C12000-22 Digital slide scanner with 0.23 micron/pixel.
Baseline characteristics for the patients included in this study. Age: age in years at the time of inclusion. Note that we report the data for 56 patients. As explained in the Methods section under ‘Patient population’, plaque-material are considered ‘waste biomaterial’ and are allowed to be used without any personal information. In this study we also considered this ‘waste biomaterial’, hence no clinical information, e.g., age, is given here nor when sharing data of the relevant samples (n=3).
Acknowledgements
We also acknowledge support from the Netherlands CardioVascular Research Initiative of the Netherlands Heart Foundation (CVON 2011/B019 and CVON 2017-20: Generating the best evidence-based pharmaceutical targets for atherosclerosis [GENIUS I&II]), the ERA-CVD program ‘druggable-MI-targets’ (grant no: 01KL1802), and the Chan Zuckerberg Initiative Foundation Data Insights program (grant no: DI-092).
We would also like to thank all the (former) employees involved in the Athero-Express Biobank Study of the Departments of Surgery of the St. Antonius Hospital Nieuwegein and University Medical Center Utrecht for their continuing work. We would like to thank (in no particular order) Marijke Linschoten, Arjan Samani, Petra H. Homoed-van der Kraak, Tim Bezemer, Tim van de Kerkhof, Joyce Vrijenhoek, Evelyn Velema, Ben van Middelaar, Sander Reukema, Robin Reijers, Joëlle van Bennekom, and Bas Nelissen. Lastly, we would like to thank all participants of the Athero-Express Biobank Study; without you these studies would not be possible.
