ABSTRACT
Importance Extranodal extension (pENE) is a critical prognostic factor in oropharyngeal cancer (OPC) that drives therapeutic disposition. Determination of pENE from radiological imaging has been associated with high inter-observer variability. However, the impact of clinician specialty on human observer performance of imaging-detected extranodal extension (iENE) remains poorly understood.
Objective To characterize the impact of clinician specialty on the accuracy of pre-operative iENE in human papillomavirus-positive (HPV+) OPC using computed tomography (CT) images.
Design, Setting, and Participants This prospective observational human performance study analyzed pre-therapy CT images from 24 HPV+ OPC patients, with duplication of 6 scans (n=30) of which 21 were pathologically confirmed pENE. Thirty-four expert observers, including 11 radiologists, 12 surgeons, and 11 radiation oncologists, independently assessed these scans for iENE and reported human-detected radiologic criteria and observer confidence.
Main Outcomes and Measures The primary outcomes included accuracy, sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and Brier score for each physician, compared to ground-truth pENE. The significance of radiographic signs for prediction of pENE were determined through logistic regression analysis. Fleiss’ kappa measured interobserver agreement, and Hanley-MacNeil AUC discrimination testing.
Results Median accuracy across all specialties was 0.57 (95%CI 0.39 to 0.73), with no specialty showing discriminate performance greater than random estimation (median AUC 0.64, 95%CI 0.44 to 0.83). Significant differences between radiologists and surgeons in Brier scores (0.33 vs. 0.26, p < 0.01), radiation oncologists and surgeons in sensitivity (0.48 vs. 0.69, p > 0.1), and radiation oncologists and radiologists/surgeons in specificity (0.89 vs. 0.56, p > 0.1). Indistinct capsular contour and nodal necrosis were significant predictors of correct pENE status among all specialties. Interobserver agreement was weak for all the radiographic criteria, regardless of specialty (κ<0.6).
Conclusions and Relevance Multiobserver testing shows physician discrimination of HPV+OPC pENE on pre-operative CT remains non-different than blind guessing, with high inter-rater variability and low diagnostic accuracy, regardless of clinician specialty. While minor differences in diagnostic performance among specialties are noted, they do not significantly affect the overall poor agreement and discrimination rates observed. The findings underscore the need for further research into automated detection systems or enhanced imaging techniques to improve the accuracy and reliability of iENE assessments in clinical practice.
Competing Interest Statement
Dr. Fuller has received unrelated direct industry grant/in-kind support, honoraria, and travel funding from Elekta AB; honoraria, and travel funding from Philips Medical Systems; and honoraria, and travel funding from Varian/Siemens Healthineers. Dr. Fuller has unrelated licensing/royalties from Kallisio, Inc. Dr. Sandulache is a consultant for, and equity holder in, Femtovox Inc (unrelated to current work).
Funding Statement
Kareem A. Wahid is supported by the Dr. John J. Kopchick Fellowship through The University of Texas MD Anderson UTHealth Graduate School of Biomedical Sciences, the American Legion Auxiliary Fellowship in Cancer Research, and an NIH/National Institute for Dental and Craniofacial Research (NIDCR) F31 fellowship (F31DE031502) and an Image Guided Cancer Therapy (IGCT) T32 Training Program Fellowship (T32CA261856). Mohamed A. Naser receives funds from NIH/NIDCR R03 grant (R03DE033550). Clifton D. Fuller receives related grant support from the NIH/NCI Cancer Center Support Grant (CCSG) Image-Guided Biologically-Informed Therapy (IDBT) Program (P30CA016672) as well as additional unrelated salary/effort support from NIH institutes. Dr. Fuller receives grant and infrastructure support from MD Anderson Cancer Center via: the Charles and Daneen Stiefel Center for Head and Neck Cancer Oropharyngeal Cancer Research Program and the Program in Image-guided Cancer Therapy. The work of Joel Jaskari, Jaakko Sahlsten, and Kimmo K. Kaski was supported in part by the Academy of Finland under Project 345449. Antti Mäkitie is supported by the Finska Läkaresällskapet. Benjamin H. Kann is supported by an NIH/National Institute for Dental and Craniofacial Research (NIDCR) K08 Grant (K08DE030216). Jussi Hirvonen receives funding from the Sigrid Jusélius Foundation. Jeffrey Guenette was supported in part by the Care and Equity in Radiology Research Academic Fellowship through the Association of University Radiologists and by a National Institute of Biomedical Imaging and Bioengineering (NIBIB) K08 Grant (K08EB034299).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
IRB RCR03-0800 and PA19-0491 of The University of Texas MD Anderson Cancer Center gave ethical approval for this work
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
In this revised version, the authors provide updates and refinements to enhance the clarity, accuracy, and overall quality of the analyses and interpretations. Key revisions are outlined as follows: 1- Abstract Revision: The abstract has been restructured to better reflect the study's objectives, methods, and findings. An updated visual abstract was also added for a more comprehensive visual summary. 2-The introduction has been expanded to provide a more comprehensive background on the importance of pENE in OPC, current diagnostic challenges, and the rationale for investigating differences in observer performance among various clinical specialties. 3- The methods section has been elaborated to provide clearer details on the study design, statistical analysis, and reporting guidelines. Additional information on image processing and the survey instrument was added for better reproducibility. 4- The results section has been supplemented to present performance metrics more clearly. Significant differences between specialties in accuracy, sensitivity, specificity, and other metrics were highlighted. Detailed tables and figures were included to illustrate these findings. 5- The discussion has been extended to interpret the results in the context of existing literature. The implications for clinical practice and the potential for automated detection systems to improve diagnostic performance were discussed.
Data Availability
Raw data are available from authors upon reasonable request. Anonymized datasheets used for analysis will be made publicly available on Figshare (DOI = 10.6084/m9.figshare.22177574) after manuscript acceptance in peer reviewed journal.
