“E Pluribus Unum”: Prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology (C3RO) Crowdsourced Initiative for Multi-Observer Segmentation

Background and Significance

Contouring, also referred to as delineation or segmentation, of regions of interest (ROIs) on medical imaging is a crucial aspect of radiation treatment planning, and has been reported as the largest single source of system uncertainty in radiotherapy [1], especially in the sense that, intrinsically, there is often no “ground truth” for absolute determination of patient-specific segmentation accuracy [2]. Manual definition and annotation of target volumes and organs-at-risk (OARs) is subject to considerable interobserver variability, even among experts [3,4], leading to inconsistent contour quality [5,6], which have been correlated with disease control decrement and increased toxicity [7–9]. Efforts to reduce manual segmentation variation have included consensus guidelines [10,11], which generally include a benchmark “gold standard” contour curated by one expert for clinical use or through a single simultaneous truth and performance level estimation (STAPLE)-consensus derived by an interdisciplinary expert panel [2]. Several studies have demonstrated that the use of contouring atlases can reduce variation in contouring[12–16], but use is limited in routine practice. More recently, Zhang et al. have shown the addition of a radiation anatomist could also reduce contour variation [17], but these efforts are still exploratory.

Auto-segmentation, broadly defined as the automated generation of contours on a digital image by a computer algorithm, has emerged as an avenue to decrease contour variability and thereby improve standardization. While automated contouring methods evolve, a significant challenge in the development of auto-segmentation algorithms is the relative paucity of curated multi-expert observer datasets sufficiently large to train machine learning models, e.g., deep learning approaches [18]. This is particularly true for disease sites such as the head and neck (H&N), which have demonstrated high interobserver segmentation variability [19,20].

As such, our team developed the Contouring Collaborative for Consensus in Radiation Oncology (C3RO), the first public crowdsourced challenge to engage radiation oncologists across various expertise levels in cloud-based image-segmentation in multiple disease sites. We sought to: 1) characterize the variability in radiation oncology segmentation performance across multiple levels of expertise compared to aggregated expert performance as “gold standard”, 2) determine whether aggregate or “composite” segmentations generated from non-experts could meet or exceed individual expert performance acceptability, and 3) examine the performance dynamics of consensus segmentation generation using a variable number of non-experts required to generate acceptable segmentation priors. An overview of our study is shown in Figure 1.

• Download figure
• Open in new tab

Figure 1.

Study workflow overview. Radiotherapy planning cases across a variety of disease sites were used to crowdsource non-expert and expert region of interest (ROI) contours. These contours were then investigated to determine interobserver variability and used in consensus segmentation experiments. Abbreviations: H&N = head and neck, GYN = gynecology, GI = gastrointestinal, STAPLE = simultaneous truth and performance level estimation algorithm.

Methods

Consensus Methods

The simultaneous truth and performance level estimation (STAPLE) algorithm [2], a well-validated and widely implemented consensus segmentation method based on weighted probabilistic estimation, was utilized to generate consensus multi-observer ROIs for this analysis. We utilized the SimpleITK [24] STAPLE implementation with a threshold value of 0.95. An example of a consensus segmentation generated from a set of expert segmentations in Figure 2.

• Download figure
• Open in new tab

Figure 2.

Expert segmentations of the left parotid gland in the head and neck case (left) and corresponding simultaneous truth and performance level estimation (STAPLE) consensus segmentation (right).

Results

As of August 2022, C3RO had 1,026 unique registrants, 221 of whom contoured at least one case. Among the participants who contoured, 127 (57%) identify as male and 93 (42%) identify as female. Participant race and ethnic backgrounds are as follows: 96 (43%) White; 80 (36%) Asian or Pacific Islander; and 22 (10%) Hispanic, Latino, or Spanish origin; 7 (3%) Black. Only 52 (24%) of participants are from the United States, while 169 (76%) of the participants are international. Most, 169 (76%), participants are practicing radiation oncologists, while 40 (18%) are resident physicians, 7 (3%) are radiation therapists, and 1 (<1%) is a medical physicist. The median (IQR) years of experience is 5 (3, 10) for attending physicians after residency and is 3 (2, 4) for resident physician year in residency. Most, 146 (66%), participants work in an academic setting or are affiliated with a University, 50 (23%) work in a non-academic hospital, and 21 (10%) are part of private practice. Participant characteristics can be found in Table 2.

Interobserver variability

Interobserver variability of non-experts and experts based on pairwise segmentation comparisons are shown in Figure 3. For the breast case, the interobserver DSC of 3/7 ROIs (CTV_Ax, CTV_IMN, and CTV_Sclav_LN) and interobserver SDSC of 2/7 ROIs (CTV_Ax and CTV_IMN) were significantly higher for experts vs. non-experts. For the sarcoma case, the interobserver DSC and SDSC of 1/3 ROIs (GTV) was significantly higher for experts vs. non-experts. For the H&N case, the interobserver DSC and SDSC of 10/11 ROIs (GTVp, GTVn, CTV1, CTV2, Brainstem, Glnd_Submand_L, Glnd_Submand_R, Musc_Constrict, parotid_l, parotid_r) were significantly higher for experts vs. non-experts. For the GYN case, the interobserver DSC and SDSC of 2/4 ROIs (CTVn_4500, Bowel_Small) were significantly higher for experts vs. non-experts. For the GI case, the interobserver DSC of 2/3 ROIs (CTV_4500, CTV_5400) was significantly higher for experts vs. non-experts. The remaining ROIs for all cases were non-significantly different between experts and non-experts. Volumetric comparisons for ROIs between non-experts and experts are shown in Appendix E; only 3 ROIs were significantly different between non-experts and experts, namely, the breast case CTV_IMN, the sarcoma case GTV, and the H&N case Parotid_L.

• Download figure
• Open in new tab

Figure 3.

Interobserver variability based on pairwise segmentation comparisons for radiation oncologists of varying expertise (non-expert, expert). Breast, sarcoma, H&N, GYN, and GI cases are shown in the 1st, 2nd, 3rd, 4th, and 5th rows, respectively. DSC and surface DSC metrics shown in left and right panels, respectively. Stars above plot indicate Mann Whitney U test level of significance: ns: p > 0.05; *: 0.01 < p <= 0.05; **: 0.001 < p <= 0.01; ***: 0.0001 < p <= 0.001; ****: p <= 0.0001. Abbreviations: H&N = head and neck, GYN = gynecology, _l = left, _r = right, gtv = gross tumor volume, ctv = clinical target volume.

STAPLE comparisons

Comparisons of consensus segmentations for all non-expert segmentations vs. consensus segmentations for all expert segmentations are shown in Figure 4. For the breast case, non-expert consensus segmentations for all 7 ROIs crossed the expert interobserver values for both DSC and SDSC. For the sarcoma case, non-expert consensus segmentations for 2 out of 3 ROIs (GTV, CTV) crossed the expert interobserver values for both DSC and SDSC. For the H&N case, non-expert consensus segmentations for 9 out of 11 ROIs (GTVp, GTVn, CTV2, Glnd_Submand_L, Glnd_Submand_R, Larynx, Musc_Constrict, parotid_l, parotid_r) crossed the expert interobserver DSC, while 7 out of 11 ROIs (GTVn, CTV2, Glnd_Submand_L, Larynx, Musc_Constrict, parotid_l, parotid_r) crossed the expert interobserver SDSC. For the GYN case, non-expert consensus segmentations for all 4 ROIs crossed the expert interobserver values for both DSC and SDSC. For the GI case, non-expert consensus segmentations for 2 out of 3 ROIs (CTV_4500, Bag_Bowel) crossed the interobserver DSC, while 2 out of 3 ROIs (CTV_4500, CTV_5400) crossed the expert interobserver SDSC.

• Download figure
• Open in new tab

Figure 4.

Dice similarity coefficient (DSC) and surface DSC (SDSC) values comparing non-expert STAPLE using maximum number of available cases to expert STAPLE. Breast, sarcoma, H&N, GYN, and GI cases are shown in the 1st, 2nd, 3rd, 4th, and 5th rows, respectively. DSC and SDSC metrics shown in left and right panels, respectively. Black dotted lines indicate median expert interobserver value for that metric. Abbreviations: H&N = head and neck, GYN = gynecology, _l = left, _r = right, gtv = gross tumor volume, ctv = clinical target volume.

STAPLE visual comparisons

We visually investigated 1 ROI in the H&N case which exhibited outlier behavior, namely CTV1. For both DSC and SDSC, the non-expert STAPLE of CTV1 was unable to cross the corresponding expert interobserver values. As shown in Figure 5, the expert STAPLE generally led to a more conservative estimate of the ROI, compared to the non-expert STAPLE which covered a greater area. For completeness, we also show CTV2 for both experts and non-experts, which also showed more conservative estimates for experts vs. non-experts.

• Download figure
• Open in new tab

Figure 5.

Expert STAPLE (green) and non-expert STAPLE (red) for CTV1 (left) and CTV2 (right) for head and neck case.

Non-expert STAPLE bootstrap Experiments

Non-expert STAPLE bootstrap experiments for the breast, sarcoma, H&N, GYN, and GI cases are shown in Figure 6. For the breast case, expert interobserver DSC was crossed between a minimum 2-4 observers across the various ROIs; the smallest minimum number of observers (2) was achieved for BrachialPlex_L and Heart, while the largest minimum number of observers (4) was achieved for CTV_Ax and CTV_IMN. For the sarcoma case, expert interobserver DSC was crossed between a minimum 3-5 observers across the various ROIs; the smallest minimum number of observers (3) was achieved for genitals, while the largest minimum number of observers (5) was achieved for CTV. For the H&N case, expert interobserver DSC was crossed between a minimum 3-5 observers across the various ROIs; the smallest minimum number of observers (3) was achieved for GTVn, Brainstem, Glnd_Submand_L, Glnd_Submand_R, and Larynx, while the largest minimum number of observers (5) was achieved for musc_constric_all and parotid_l. For the GYN case, expert interobserver DSC was crossed between a minimum 3-5 observers across the various ROIs; the smallest minimum number of observers (3) was achieved for GTVn, while the largest minimum number of observers (5) was achieved for CTVn_4500. For the GI case, expert interobserver DSC was crossed using a minimum 3 observers for all ROIs. Heatmap representations of bootstrap experiments can be found in Appendix D, Figure D18. Of note, the following ROIs showed non-saturating performance with an increasing number of non-experts used in the consensus segmentation: breast (BrachialPlex_L: highest non-expert DSC = 0.63, non-expert DSC using all observers = 0.42), sarcoma (genitals: highest non-expert DSC = 0.83, non-expert DSC using all observers = 0.60), H&N (GTVp: highest non-expert DSC = 0.87, non-expert DSC using all observers = 0.80; CTV1: highest non-expert DSC = 0.86, non-expert DSC using all observers = 0.64), GYN (GTVn: highest non-expert DSC = 0.90, non-expert DSC using all observers = 0.88), GI (CTV_5400: highest non-expert DSC = 0.78, non-expert DSC using all observers = 0.58; Bag_Bowel: highest non-expert DSC = 0.85, non-expert DSC using all observers = 0.74). Bootstrap experiments for additional metrics in line plot format can be found in Appendix D.

• Download figure
• Open in new tab

Figure 6.

Consensus segmentation bootstrap experiments. Pink, red, blue, purple, and green plots correspond to breast, sarcoma, head and neck, gynecologic, and gastrointestinal regions of interest, respectively. Black dotted lines indicate median expert interobserver dice similarity coefficient (DSC) for a corresponding region of interest. Gray dotted lines indicate DSC performance using the maximum number of non-experts in the consensus segmentation.

Discussion

In this study, we have systematically investigated the difference between non-experts and experts in contouring ROIs for several disease sites using various evaluation metrics. We have probed the inherent interobserver variability within non-experts and experts and determined several ROIs have better agreement when contoured by experts. Consensus segmentation experiments reveal that most consensus expert ROI contours can be roughly approximated using non-expert segmentations, which cross expert interobserver group variability performance thresholds. Our results provide justification towards using large-scale non-expert contours for “gold-standard” segmentation data in the absence of multiple expert “ground-truth” data availability, and glean insight into the behavior of consensus contours across a large number of observer inputs. While crowdsourcing is common in medical image analysis [29], there have been few studies evaluating the use of crowdsourcing for contour quality. To our knowledge, this is the largest study characterizing segmentation performance across multiple physician observers, and the first study to investigate crowdsourced contour performance in the context of radiation oncology workflows.

Our interobserver variability experiments demonstrate that several ROIs across the various cases have higher interobserver agreement for experts compared to those of non-experts for both volumetric and surface distance metrics. Generally, the interobserver variability did not vary significantly among the OAR structures for most cases. Key outliers to these trends were the majority of H&N case OAR ROIs and the Bowel_Small ROI of the GYN case. Analogously, target volumes tended to be among the ROIs that were significantly different between non-expert and expert interobserver variability. It is well known that tumor-related tissues are inherently more heterogeneous than healthy tissues. As such, our results may be explained by the potential higher subjectivity of target volume contours compared to those of OARs. A previous study by Cardenas et al. investigating large-scale multi-observer segmentation in H&N cancer using magnetic resonance imaging supports our results, whereby target volumes demonstrated particularly low agreement between observers compared to OARs [30]. Nonetheless, it warrants highlighting that certain cases exhibited a greater predilection for improved contour consistency in experts compared to those of non-experts. This is most apparent in the H&N case, where the vast majority of ROIs (10/11) showed higher agreement for experts compared to those of non-experts. These findings are congruent with previous literature indicating that H&N is a particularly challenging disease site for physician-based contouring [9,19,31].

Our initial investigations comparing STAPLE consensus segmentations using all non-expert observers against the STAPLE consensus segmentation using all expert observers revealed that non-expert STAPLE contours could cross expert interobserver variability for most ROIs; generally, there was strong agreement between results for volumetric and surface distance metrics. However, a few key outliers were unable to cross interobserver variability, namely, the genitals ROI (DSC and SDSC) for the sarcoma case, the GTVp (SDSC only), CTV1 (DSC and SDSC), Brainstem (SDSC only), and Glnd_Submand_L (SDSC only) ROIs for the H&N case, and the CTV_5400 (DSC only) and Bag_Bowel (SDSC only) ROIs for the GI case. Interestingly, unlike the interobserver variability analysis, where target volumes tended to have greater variability among non-experts, this trend did not necessarily translate when considering consensus segmentations of all the non-expert observers. Importantly, one particular ROI that had a large degree of difference between non-expert and expert STAPLE consensus contours was CTV1 of the H&N case. Upon visual investigation, the difference between non-expert and expert STAPLE segmentations was likely due to considerable non-expert uncertainty stemming from the inclusion of two individual subregions (GTVn and GTVp) in addition to areas of microscopic tumor spread, thereby leading to a larger consensus segmentation for the non-experts. These differences became less apparent when asking non-experts to contour CTV2, as the incorporation of non-ambiguous tissue (lymph node levels) seemingly increased the conformity between observers. It is worthy to note that in Cardenas et al. individual differences between GTV and CTV ROIs were minimal and are likely attributed to separate CTVs generated for primary and nodal tumors [30], as opposed to our study where a single CTV was generated combining both primary and nodal volumes.

Having confirmed that non-expert consensus segmentations could approximate expert consensus segmentations to a reasonable degree by crossing expert interobserver variability cutoffs, we sought to determine how the dynamics of segmentation performance were affected by the number of non-experts used in the STAPLE algorithm. For most ROIs, there was a general trend that on the order of 2-5 non-experts were needed to cross expert interobserver variability. Our results are congruent with previous literature on crowdsourcing labels for pathological patterns in lung imaging, where a limited number of observers could be combined through consensus methods to match reference repeatability [32]. Consistent with trends observed in our interobserver variability experiments, the ROIs that required the greatest number of non-experts to cross expert interobserver variability cutoffs were often target volumes, while the ROIs that required the least number of non-experts were OARs. The majority of ROIs exhibited a maximum DSC value, i.e., performance saturation, at a certain optimal number of non-experts used in the STAPLE algorithm, which then plateaued and maintained this high performance up to the maximum number of observers used. However, a small number of ROIs exhibited non-saturating performance effects, where after maximum performance was achieved the addition of a greater number of observers in the STAPLE algorithm decreased performance, often precipitously. As before, these tended to be target volumes where a large degree of heterogeneity between the non-experts was expected, but also included a few OARs that would be considered particularly challenging because of heterogeneity in visual interpretation, e.g., brachial plexus (breast case) and genitals (sarcoma case).

In the medical image segmentation space, several studies have been conducted on the use of “noisy” labels for model training, with mixed results. Within radiation oncology, one study in particular has shown that for at least OAR contours, deep learning may be robust to noisy segmentations [33]. A similar study investigating cardiac segmentation on ultrasound found that the training of deep learning models with novice data was not significantly different from deep learning models trained with expert data [34]. Additionally, the authors show average DSC for expert-expert and novice-expert segmentations show that no statistical difference is found between the variability of annotators in several of the annotations of interest, results which are echoed in a recent study investigating crowdsourcing for liver tumor segmentation where the quality of annotations was not statistically significantly different four distinct groups [35]. Contrary to the previous studies, Wesemeyer et al. demonstrate a tradeoff between quality and quantity for deep learning segmentation performance [36]. As we demonstrate in our study, expert consistency may be better than non-expert consistency for some radiotherapy-related ROIs, particularly for H&N imaging. Therefore, there is still likely a need to generate “expert-level” gold-standard contours in training radiotherapy-related deep learning models, at least for select cases or structures. Our study demonstrates that the use of consensus contours using non-experts may be a reasonable approximation to gold-standard expert contours using a relatively small number of observers, and may exhibit particular utility in scenarios where auto-segmentation models require large amounts of data but only a limited number of experts are available to provide segmentations.

Our study has several limitations. Firstly, we have only investigated one case per disease site in a single imaging modality (computed tomography). Therefore, our results may not necessarily generalize to arbitrary cases in different imaging modalities (e.g., magnetic resonance imaging, positron emission tomography, etc.), particularly for target volumes which can exhibit significant heterogeneity between cases and modalities. However, given that computed tomography is the current gold-standard for radiotherapy planning, we believe our results are of significant interest to the radiation oncology community. There was no contouring protocol provided to the participants for reference in this challenge, which may have reduced optimal performance. To evaluate contours, we utilized geometric indices, which are not well correlated with clinically meaningful endpoints; rather, a multi-domain approach including dosimetric indices and clinical input has been shown to be the best method to evaluate auto-segmentation [25]. An additional limitation of our study is that we have stratified physician expertise based on subjective criteria. Herein, we have defined an expert as an individual who is recognized in their field and/or contributed to consensus guideline generation. While there may be additional methodology to stratify expertise, we have chosen this method as it separates physicians based on perceived familiarity with standardized guidelines. Nonetheless, despite this limitation, we show a difference between this stratification of experts and non-experts. More studies are needed to determine objective criteria for expertise. We fully acknowledge this may mischaracterize some individuals into “non-expert” categories erroneously, even if they have significant familiarity with guidelines, but plan to investigate alternative stratifications based on additional objective criteria in future studies. Moreover, our current analysis only investigates radiation oncologists who have at some level completed formal training. We have not investigated novice observers, where observers may have no formal education in medical anatomy or segmentation. Future studies should investigate novice observers as these labels would potentially be the most cost effective to obtain and may also be able to approximate the current gold-standard.

Importantly, our study provides a large high-quality curated dataset that can act as a reference for future studies on interobserver contour variability and auto-segmentation in radiation oncology workflows. Moreover, we publicly distribute our raw imaging data and open-source our analysis pipelines so the community can investigate these claims further. Finally, our results highlight the differences in non-expert and expert contours, which can be further leveraged to create educational tools for trainee segmentation quality control.

Conclusions

In summary, using five distinct disease sites (breast, sarcoma, H&N, GYN, and GI), we have systematically investigated differences in contour quality between non-expert and expert radiation oncologist observers in target volumes and OARs. Overall, there was a general trend towards experts providing more consistent segmentations in terms of pairwise DSC and SDSC for a variety of ROIs. Moreover, we showed that using the STAPLE algorithm, consensus contours could be generated from non-experts that approximate gold-standard expert segmentations to a reasonable degree (crossing expert interobserver variability) for most ROIs; some target volumes were unable to be approximated readily. Finally, we experimented with a variable number of non-experts in generating consensus contours and demonstrated for most ROIs 2-5 non-experts is sufficient to cross expert interobserver variability, though specific attention should be paid to ROIs such as target volumes and complex OARs which exhibit decreased performance as more observers are added to the consensus segmentation. Our study acts as a potential reference for the characterization of interobserver variability and use of consensus contours in future AI-related radiotherapy applications. Future work will include the investigation of a greater number of disease sites, cases, imaging modalities, and levels of expertise.