Clinical Acceptability of Automatically Generated Lymph Node Levels and Structures of Deglutition and Mastication for Head and Neck Cancer Patient Radiation Treatment Planning

Introduction

Artificial intelligence-based auto-segmentation models are being adopted in clinical practice within radiation oncology. The benefits of such approaches are well known and include time-savings^{1, 2}, decreases in variability³, and quality assurance applications ^4–6. Such advances are pertinent to the head and neck, as delineation accuracy of OARs (organs at risk) and targets are limited by interobserver and trial protocol variability^7–9. Within the anatomic site of head and neck, auto-segmentation models developed through deep learning has resulted in a myriad of contouring approaches^{4, 10–17}. Many of these models focus on contouring OARs that can be delineated from radiotherapy simulation CT scans. Commercial models for fully automated target segmentation in the head and neck are not yet available, but research in this area is gaining momentum.

Of recent interest is the auto-segmentation of the low-risk, or elective clinical target volume (CTV). The low-risk CTV is comprised of anatomically-defined lymph node-containing regions (“lymph node levels”) that are at risk of metastatic spread, though possess no clinical or radiographic evidence of disease at the time of treatment. The set of lymph node levels selected for inclusion in the low-risk CTV is based on lymphatic drainage patterns from the location of the primary tumor. Generally accepted volumes based on common tumor locations are well documented in consensus guidelines and contouring resources for the head and neck^{9, 18}. Automatic CT-based segmentation of these lymph node levels (e.g. I-V) is achievable and has been demonstrated by numerous works^{10, 19–25}. Our clinic previously integrated a deep-learning based approach to contour elective lymph node levels in CT scans¹⁰. This approach groups nodal levels into families (e.g. IA-V, IB-V, II-IV, and retropharyngeal [RP]) so that elective CTVs can be quickly constructed using Boolean algebra for use in manual and automatic treatment planning. However, one significant challenge in the field of target segmentation is keeping pace with changes in clinical practice. Sources of such changes can be driven by changes in contouring guidelines, improvements in image-guidance technology, or evolving evidence that alters our understanding of the balance between toxicity and tumor control^{26, 27}. Strijbis et also noted in their work on automated segmentation of levels I-V that contours produced by Cardenas et al are generous, resembling their institution’s PTVs²⁵. Based on physician feedback and changes in clinical practice, we sought to develop a new auto-segmentation model that more accurately reflects the narrower treatment volumes utilized in our clinic’s practice today.

Proximal to these CTVs are essential structures which enable deglutition (i.e. swallowing) and mastication (i.e. chewing). When the functions of these structures are compromised, side effects such as dysphagia can occur. Dysphagia is one of the strongest determinants of quality of life following radiation therapy and can affect many physical, mental, and social components of life^28–31. Many works have well characterized the dose-effect relationships for swallowing and chewing structures^{28, 32, 33}. Although publicly available models and repositories exist for OARs in the head and neck, to our knowledge there exists no such repository of swallowing structures. Teguh et al were the last to clinically validate an approach to segment lymph levels and swallowing structures in the head and neck in one combined work, but did so with atlas-based auto-segmentation¹⁹. Many other authors have studied swallowing and chewing structure segmentation, but only include some of the major swallowing and chewing structures that can be visualized on CT. A comprehensive auto-contouring approach to generate these structures could provide a means for reliable and efficient assessment of dose response studies—especially since manual delineation of these many structures is extremely labor intensive.

The purpose of this work is to present a straightforward approach that can efficiently and effectively create and validate a clinical segmentation tool for 1) individual lymph node levels and 2) swallowing and chewing structures in the head and neck. For the lymph nodes, the hypothesis of this work is that our approach will result in contours that will be as clinically acceptable as physicians’ own manual contours and be preferred at rates equal or superior to manual contours. Such an auto-contouring model could eliminate sources of intra-physician variability present in elective target contouring if used clinically. This will be tested using two double blinded studies to score physician preference and contour quality (for manual vs AI contours). For the swallowing structures, we use a simpler evaluation approach (i.e. no blinded studies) to gauge clinical acceptability of resulting AI contours.

In a busy practice with limited physician time, data scarcity can be a challenge for clinics that are wanting to develop their own tools or perform clinical research. To this end, we have published the first clinically validated dataset of cervical lymph nodes (XXXX) and swallowing and chewing structures (XXXX) on The Cancer Imaging Archive (TCIA).

Materials and Methods

The MD Anderson Cancer Center has 10 physicians 11 physicists, and 8 dosimetrists who specialize in head and neck cancer at its main hospital. Our treatment planning process for the head and neck uses auto-contouring for organs-at-risk⁴ and lymph node level families¹⁰ for every patient (> 100 per month). The following subsections outline the data curation, model training, quantitative evaluation, and design of blinded studies. This work followed the recommendations by Baroudi et al which suggest guidance for quantitative (using overlap and distance metrics) and qualitative evaluation (using physician review) of clinical acceptability³⁴. Thus, both quantitative and qualitative review metrics are reported from the literature for lymph node segmentation as well as swallowing and chewing structure segmentation.

Quantitative analysis of lymph node level, chewing, and swallowing structures

Dice Similarity Coefficient (DSC) and 95^th percentile Hausdorff (HD95th) distance were used as quantitative performance metrics between the ground truth contours and the predicted contours.

For point sets A and B, representing 3-dimensional segmentation volumes, the DSC and HD95^th are defined in Equations 1-3. |A| and |B| denote each 3-dimensional segmentation volume; S₁ and S₂ are the surfaces from A and B, respectively.

Results

Physician review of AI-based swallowing and chewing structure segmentations

Of the 340 contours reviewed, 100% were found to be 3 or greater (minor edits - use as is) as noted in Table 3. 96% percent of contours were scored as 4 or 5 (stylistic differences, use-as-is). Thyroid cartilage, glottic area, larynx, buccinators, digastrics, masseters, mgh complex, and pterygoids had perfect scores (5’s) across all patients. Four common issues were identified (1) The genioglossus would contour into the mgh complex when transitioning, (2) The anterior digastrics overcontour into the mgh complex, (3) The anterior digastrics would overcontour posteriorly, and (4) Transitions between the superior constrictor and medial constrictor, medial constrictor and inferior constrictor, and inferior constrictor and cricopharyngeus would be incomplete by missing a slice between structures or by splitting into islands at the transition. In general, the algorithm performed well in the presence of dental artifact (Fig A1). Examples of minor edits needing to be performed are featured in Figure 6 in the presence of tumor or dental stent which distorts the tongue.

• Download figure
• Open in new tab

Figure 1

| Quantitative metrics for lymph node level segmentation. dsc, dice similarity coefficient; hd95, 95^th percentile Hausdorff distance

View this table:

• View inline
• View popup
• Download powerpoint

Table 3

| Physician scoring of clinical acceptability for swallowing and chewing structures

Discussion

This work has demonstrated how a publicly available deep learning model trained with expertly curated data (n = 32 lymph node dataset, n = 78 swallowing/chewing structure dataset) can result in a practical segmentation tool that is both accurate and clinically acceptable. Furthermore, this work has resulted in two published datasets (in The Cancer Imaging Archive) that are available for clinical researchers to reproduce or build upon the field of head and neck CT-based tissue segmentation. To our knowledge, these two datasets represent the first publicly available datasets for swallowing and chewing structures and lymph nodes contoured on non-contrast CT, both with physician scores of clinical acceptability.

One early work in lymph node and swallowing structures segmentation in the head and neck was by Teguh et al which obtained a DSC of 0.73 on average for lymph node levels I-V and 0.50–0.71 DSC for swallowing and chewing structures¹⁹. Since then, multiple authors have continued to build more accurate models using deep learning, while simultaneously increasing the number of structures predicted by deep learning models (Tables 4 and 5). While various works such as van der Veen and Weissmann et al reported qualitative evaluations (e.g. radiation oncologist scores) and quantitative evaluations (e.g. DSC, HD) for lymph node model contours, few authors reported what proportion of deep learning contours require edits before the contours can be used clinically (Table 4). For swallowing and chewing structures, most authors have reported what percentage of their test set contours require edits before clinical use (Table 5). These rates have ranged from 100% to 18% for various structures with DSC’s ranging from 0.5 to at most 0.91 on average or median. In addition to accuracy and clinical acceptability, previous works have extensively studied human interobserver variability in lymph node level contouring and have quantified it in terms of DSC^{23, 24}. Weissman et al demonstrated that when physicians re-contoured their own volumes, DSC was 0.77 on average²⁴. Overall, they reported that the accuracy of their deep learning approach performed with 0.78 DSC on average across 20 nodal levels and made their deep learning model publicly available.

Regarding the lymph nodes segmentation performance of our approach, quantitative analysis of the lymph node level segmentations indicated that DSC ranged from 0.77 to 0.89 across all levels and is comparable to other recent approaches in the literature (Table 4). Regarding qualitative evaluation, a blinded study investigating physician preference indicated that the AI contours were superior to or equally preferred to the manual contours at rates ranging from 75% to 91%. In addition, a blinded study of clinical acceptability indicated that there was no significant difference in clinical acceptability between manual contours and AI contours for levels IA, IB, II, III, IV, and V, with greater clinical acceptability demonstrated for AI-generated RP contours. Furthermore, only 8% of AI lymph node contours require edits for clinical use as determined by our team of sub-specialized head and neck radiation oncologists. Thus, the hypothesis of this work can be accepted, since 1) there is no difference in clinical acceptability between manual and AI contours and 2) the AI contours were superior to or equally preferred to the manual contours. Although not directly comparable, our average DSC’s in the nodal test set are all greater than previously reported measures of interobserver variability mentioned above. Anecdotally, this may suggest that our model accuracy has similar or smaller levels of variance compared to human contouring in our practice since there was no preference for manual vs AI contours (p <0.01) for all but one nodal level. As is performed in clinical practice, the lymph node model learned to seamlessly abut adjacent nodal levels (i.e. no gaps). However, this abutment did not coincide with the axial slice planes which are orthogonal to the superior-inferior direction (Figure 2B). A slice plane adjustment function was developed by Weissman et al, to post process contours so that they abut in axial planes and more closely resemble human contours which are typically contoured in axial views²⁴. While this technique resulted in better physician ratings in Weissman’s study (vs no post processing), such a technique was not applied to our testing cohort, as our lymph node contours were scored as equivalent or superior to manual contours without post processing.

• Download figure
• Open in new tab

Figure 2

| Segmentations of lymph node levels IA, IB, II, III, IV, V, and RP (columns A-G, respectively). Solid lines represent ground truth segmentations and dotted lines represent predicted segmentations.

Strijbis et al demonstrated similar quantitative lymph node model performances values to our work but used a training cohort roughly twice the size of our training cohort²⁵. However, as demonstrated by Yu et al, there are diminishing returns for increasing U-Net segmentation performance as a function of training dataset size³⁶. This is especially true when the training data (in this case CT) contain images of patients with consistent patient positioning, immobilization, and imaging protocol. Weismann et al demonstrated that a clinically acceptable lymph segmentation model could be made with the publicly available nnUnet architecture with a small cohort of patients (n = 35)²⁴. Likewise, we have found this number to be adequate for our patient population of head and neck patients which receive radiotherapy simulation CTs.

Regarding the swallowing structures segmenation performance, excellent quantitative and qualitative measures were obtained on the swallowing/chewing structures test set. Our work produced a model that segments 17 different swallowing and chewing structures, more than any work, to our knowledge, has produced. This publicly available data will be a valuable source of data for future head and neck studies. Median DSC was greater than 0.86 for all structures, and although values in the literature cannot be directly compared between different datasets, our approach yields the most accurate structures reported in terms of DSC. Furthermore, it was demonstrated that 96% percent of contours in the test set could be used clinically without edit and this percentage is much greater than any rates previously reported in the literature for swallowing and chewing structures (Table 5). Dental stents distorting tongue position (Figure 6B) and the presence of abutting tumor (Figure 6B) were common factors which necessitated minor edits. The hypothesis of this work, that swallowing and chewing AI contours are clinically acceptable, can be accepted.

One limitation of this work is that the impact on the model that variations in patient population and radiotherapy simulation protocol were not studied. Future studies would benefit from studying the effects of patient positioning, adenopathy (as noted in Fig 3A), nodal level dissection, bulky disease, and presence of contrast upon segmentation performance. In this work, training and testing cohorts for both datasets used a three-step methodology in which ground truth data was generated by a sparsely trained deep learning model or atlas-based model. Although this strategy is useful for perpetuating sparse data, it can bias the curation process if contours are not comprehensively edited after it is generated by the atlas or sparsely trained model. Lastly, not all nodal levels were included in this work, as many listed in consensus guidelines are not commonly used. Future model refinements will include a more comprehensive selection of nodal levels.

• Download figure
• Open in new tab

Figure 3

| Featured segmentations. White arrow in A indicates bulky adenopathy at level II (red). White arrows in B indicate where predicted levels III (dotted blue) and IV (dotted pink) slope in the inferior-superior direction.

• Download figure
• Open in new tab

Figure 4

| Quantitative metrics for swallowing/chewing structure segmentations. dsc, dice similarity coefficient; hd95, 95^th percentile Hausdorff distance

Median DSC for the test set (n = 20) ranged from 0.86 to 0.96 (Fig 4A); median HD95^th (in mm) ranged from 1.2 to 2.5 (Fig 4B). Median DSC was greater than 0.90 for 11/17 structures.

• Download figure
• Open in new tab

Figure 5

| Segmentations of all swallowing/chewing structures. 5A is the mid-sagittal plane that cuts through a majority of the midline structures; 5B is a sagittal plane that cuts through the buccinator and pterygoids. Solid lines represent ground truth segmentations and dotted lines represent predicted segmentations. In many of the figures, solid and dotted lines are overlapped.

• Download figure
• Open in new tab

Figure 6

| Featured segmentations for swallowing/chewing structures requiring minor edits. White arrow in A indicates the presence of tumor that is displacing local anatomy. White arrows in B indicate where the tongue is displaced by the dental stent.

These models for lymph nodes and swallowing and chewing structures are being integrated into our clinical practice at MD Anderson as well as the Radiation Planning Assistant (RPA), a web-based, FDA 510k cleared platform that provides contouring and planning to clinics with low resources. Future work will involve multi-institutional studies to evaluate robustness as well as application based-works that use these tools for clinical trial quality assurance and toxicity analyses.

Conclusion

This work demonstrated an approach to generate clinically acceptable contours for lymph node levels, swallowing structures, and chewing structures for CT-based segmentation in the head and neck. A publicly available deep learning model using data curated by sub-specialized radiation oncologists in the head and neck resulted AI contours which are, on the vast majority, clinically acceptable and highly accurate. This work has also produced two separate datasets that are the first publicly available datasets for lymph node levels and swallowing and chewing structures in the head and neck.